Evan Jones
@evanqjones
Followers
1,740
Following
726
Media
65
Statuses
347
ML @HavenStudiosInc prev SWE @PaloAltoNtwks @Twitter . Hacking on LLMs and text-to-image. Llama addict. Opinions are mine and not my employer.
Raleigh, NC
Joined March 2013
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
América
• 914590 Tweets
HAPPY BIRTHDAY JIMIN
• 538042 Tweets
España
• 469598 Tweets
Texas
• 180344 Tweets
Colón
• 92734 Tweets
#elevenallstar
• 74548 Tweets
#StarAcademy
• 72881 Tweets
Cristiano Ronaldo
• 68395 Tweets
Tigers
• 67423 Tweets
Türkiye Devleti
• 54972 Tweets
Numan
• 50249 Tweets
Alex Salmond
• 45875 Tweets
Vinicius
• 45133 Tweets
South Carolina
• 36395 Tweets
#BeterbievBivol
• 34379 Tweets
Guardians
• 31279 Tweets
Bama
• 29057 Tweets
Penn State
• 29021 Tweets
Dick Cheney
• 28879 Tweets
Ramon
• 26393 Tweets
Cleveland
• 25458 Tweets
Oklahoma
• 21406 Tweets
رونالدو
• 20375 Tweets
Lane Thomas
• 19415 Tweets
Arch
• 19093 Tweets
Clarke
• 17685 Tweets
#POLPOR
• 15573 Tweets
Leao
• 14941 Tweets
Skubal
• 14755 Tweets
Wardley
• 13006 Tweets
Tito Mboweni
• 10538 Tweets
Raising a little garden from latent space
On M2 Max, using SD 1.5, ControlNet, TripoSR, and three.js. Sped up for brevity.
44
134
1K
This is the construct. We can load anything from Chesterfields to vintage TVs. Anything we need.
Made with three.js, SD 1.5, ControlNet, and TripoSR, running on an M2 Max (generation sequences sped up for brevity).
23
135
709
Made a little scene editor/interior design tool with Stable Diffusion inpainting - running locally on M2 Max, using the Dreamshaper finetune by
@Lykon4072
32
40
582
Added brushes to my Stable Diffusion isometric editor: texture, object, and wall. It can now build a room from scratch. All running locally on an M2 Max (video at 2x for brevity).
19
55
463
Integrated Llama into my Stable Diffusion isometric editor. Now they can dream up Sims-esque shopping and object interactions. Running locally on M2.
14
26
348
AI-generated strategy game, brought to you by local Llama and Stable Diffusion
15
27
316
Published the code for my ControlNet-based 3D texture generation script for TripoSR outputs. Link below.
8
30
176
Let's dream up a Gothic tomb - with SD 1.5, ControlNet, TripoSR, three.js
4
19
162
Added people, inventory, and interactions to my llama.cpp-based RPG
8
10
143
Applying textures generated by Stable Diffusion in real time, locally, in a 3D environment
12
14
142
Texturing 3D Meshes with Stable Diffusion + ControlNet
I love the meshes coming out of TripoSR by
@StabilityAI
@tripoai
but sometimes they lose some of the detail from the original image. I wanted to (re)apply a Stable Diffusion image as a texture to regain detail. Full
2
32
139
Updated my ControlNet-based texturing script for TripoSR meshes to go all the way around the model. Still a single generation, 12 steps in this case, at ~1.5s/step on M2 Max. Link to code below.
2
16
70
Got
@StabilityAI
Stable Zero123 running on a Mac with the original zero123 code (which I've modified to be a package). The trick to getting proper outputs turned out to be setting the z distance to a constant pi / 2.
2
4
47
This is now open source! You, too, can draw forth angels from latent space. Link below
Let's dream up a Gothic tomb - with SD 1.5, ControlNet, TripoSR, three.js
4
19
162
2
6
32
I've just open-sourced my experimental adventure game running on llama.cpp in the terminal. It's rough and unfinished ... but functional!
Read on for some of the wild adventures you could take:
2
3
32
Exploring a game world generated on the fly by Code Llama, running locally on llama.cpp (loading sequences clipped)
3
2
29
Sprite generation workflow using Stable Diffusion and Stable Zero123, running on M2 Max
0
0
19
Realizing that once we solve generative game mechanics, we can remake Spore to finally live up to the original hype
Maybe it was AI-complete all along
3
0
18
Text to (partial) 3D on a Mac M2 using Stable Diffusion and Depth Anything (image gen clipped for brevity)
1
0
18
Experimenting with open-vocabulary object detection in generated scenes. Idea being to generate a scene with an image model, decompose it into semantic segments, and build the game world from that. This one is a super quick mockup of ChatGPT -> DALL-E -> YOLO World.
Basically
3
1
17
Thinking about the Roman Empire... in a little Stable Diffusion city builder
0
0
17
@yacineMTB
I was actually thinking of doing something with predicate logic as one of my next projects
1
0
14
@JungleSilicon
Yes it's inpainting. Currently the objects are first painted on the base scene (e.g., empty room) and then composited. Working on further extracting them to allow for manipulation etc.
1
1
14
LLM-driven games will shine where language is intrinsic to the gameplay. Like taking the captain's chair in a starship. Or governing by written edicts and policies. Or having to rule through your ministers.
1
0
12
Wrote up a guide for llama.cpp grammars:
1
1
11
Working on an approach for text -> multiple views on an object with just a general purpose Stable Diffusion model. Interested to see how far I can take basic inpainting, and the initial results are looking promising!
Technique in thread...
1
0
12
Latent Clay is available here:
2
0
11
its sculptor well those passions read
The hand that mocked them and the heart that fed:
And on the pedestal these words appear:
"My name is Ozymandias, King of Kings:
Look on my works, ye Mighty, and despair!"
Sculpted by M2 Max running SD 1.5, ControlNet, and TripoSR
3
0
10
Starting to experiment with SDXL. After several hours of trial and error, I happened upon a configuration that produces semi-usable game assets
0
1
11
Part of maturing as an engineer has been coming to terms with technical debt. You should build smart but do what it takes to ship now. Take on whatever debt is necessary. I've seen so many finely crafted codebases scuppered by changing business factors and product decisions.
1
1
11
Here's Llama generating a tomb adventure map on the fly when prompted a game name. The engine delegates scene layouts to the model, so it generalizes to arbitrary settings.
1
0
11
First, the mesh is simplified and raycast to a depth map using Open3D. Then, ControlNet for depth is used to paint the depth map. Finally, the image result is traced back to the mesh's UV map via the raycast results to generate & apply the texture.
3
0
10
Been playing with depth maps and I think I've finally figured out the conversion of RGB + depth -> textured mesh in Python
2
0
10
I built something like this internally at Twitter but for video captures. You passed it a Python script where each function navigated to a page and manipulated a virtual mouse, and it output a gif for each of these little scenarios.
New release of shot-scraper, my CLI tool for taking screenshots and running JavaScript scrapers from the terminal
1.4 adds support for HTTP Basic auth, custom --scale-factor shots, additional --browser-arg arguments and a fix for --interactive mode
7
19
161
1
0
8
More tests of my Stable Diffusion -> Depth Anything -> partial 3D pipeline on a Mac M2
0
0
9
I've just open sourced Latent Clay, a local-first, Mac-friendly AI world sandbox I used to create my demos the past few months. Link in the reply:
1
0
9
- Drew a tilted cube at a few rotations using the awesome 3D modeling technology of ... CSS
- Added an arrow to help guide the model
- Painted the cubes white for the mask
- Inpainted with
@Lykon4072
Dreamshaper finetune with "3d model of X at various rotations"
4
0
9
@BartronPolygon
Not customized for this task but it is an SD fine tune, it's Dreamshaper 8 by
@Lykon4072
1
1
9
Accidentally introduced Llama-generated loot to my llama.cpp-based RPG - I was too lazy to look up the items in the scene and let the model decide
0
0
9
@loveofdoing
I plan to open source pretty soon, just trying to get core functionality and then clean up code
2
0
8
Experimenting with laying out and tiling SDXL-generated game assets. Interestingly, isometric turned out to be easier than top-down - the model seems to have difficulty with overhead views.
3
0
8
@kickiniteasy
@Lykon4072
It's a pretty basic local webapp running diffusers/PyTorch (and the app itself was written in Vim 😉)
0
0
7
My kids were fighting over coloring pages so I decided to inpaint a new one with Stable Diffusion. Sharing my very sophisticated Preview dot app workflow here. The python script towards the end is basically a wrapper around diffusers that dumps iterations to the filesystem.
0
0
6
Working on a little generative RPG for llama.cpp that uses grammar to track game state and actors to manage setting and character memory
0
1
6
Thinking about making a holodeck soon. And building it in CSS would be the funniest outcome
0
1
6
Ironically this plays way better in a text adventure right now, where the latency on the action is just straight tok/s and the pauses for user input fit with the medium
I'm interested in AI-driven improvised action sequences in games. Really capture the novelty of a Bourne fight or a Bond chase. Like a goon pops out of the fridge and you're fending them off with a toaster.
0
0
2
1
0
6
@Tocelot
I've been experimenting with some of these ideas with local AI
Added people, inventory, and interactions to my llama.cpp-based RPG
8
10
143
1
0
6
@ThinkWiselyMatt
@Lykon4072
Here's what I ended up on (at 2x for brevity), probably needs some tweaks, hah. It really wanted to turn it into furniture. And yeah, want to get to characters as well. The challenge there being the animation - haven't wrapped my head around that yet.
1
0
5
@raw_works
That's the plan, but here it's just the colored meshes from TripoSR. The textures will be generated in the background after the initial mesh loads in
1
0
6
Trying out the new TripoSR single image-to-3D model from
@StabilityAI
and
@tripoai
on a Mac. As promised, it is quite good, esp. for the speed. Runs in 6s on the M2 Max, could possibly be improved further. Some initial observations in the thread:
1
1
3
Working on a new floor generation approach, and the model desperately wants to create these intricate, symmetrical floor patterns. Kind of a cool finding, could be selectively applied for effect.
Also every time I go back to floor gen I feel like I should do a 2d adventure
0
0
5
Here's where I'm at with automating multi view sprites from SD inpainting (at 2x). It's promising, but the quality and consistency isn't quite there & hitting all the angles is a challenge.
0
0
5
Finally figured out how to compose a transform matrix for a surface in a CSS 3D scene and translate screen coordinates to element coordinates (more in thread)
1
0
5
@simonw
For specifying the JSON shape, there's also `examples/json-schema-to-grammar.py` in llama.cpp that converts JSON schema to GBNF.
1
0
6
Grammar sampling is now available in whisper.cpp. We ported it over from llama.cpp and it uses the same GBNF syntax. Based on early testing, it's definitely a different beast than with a generative model - some experimentation is needed to figure out how best to apply it.
1
0
5
@MatthewBerman
Haven't decided how I'm going to release this. But I am preparing to release my older experiment with a terminal-based Llama RPG
Added people, inventory, and interactions to my llama.cpp-based RPG
8
10
143
0
0
4
@yuiseki_
1) my philosophy is local-first, so folks should be able to run privately on their own hardware but have the option to swap in hosted models as well
2) basically convenience? Also browsers are universal. My experience is more with web apps and I'm figuring out 3d as I go
0
1
4
Drawing out RPG room layouts dreamed up by Code Llama... starting to come together
0
0
4
The biggest challenges I'm facing now are:
* aligning generative models to intended goals and ethical principles
* aligning my 5 and 6 year old sons to intended goals and ethical principles
0
0
4
I implemented JSON schema for llama.cpp grammars but I'm more interested atm in lightly annotated natural language. The thought is that it might be a better fit to models' training data and more token efficient than JSON.
2
0
4
@ggerganov
Eventually it will. It generates code behind the scenes to lay out the rooms (and other game logic) so I'll constrain it to that eventually. For now I'm relying on the model following the pattern from the prompt, which it does more often than not actually.
0
0
4
Got zero123 (model for changing camera viewpoint on an object) running on a Mac M2. Required surprisingly few changes to the original CUDA-based code - it's impressive how well torch's MPS backend works. The patch is on my fork which I'll link.
1
1
4
Playing around with rendering images in the terminal using ANSI escapes in 256 colors. Using the "lower half block" trick from .
0
0
3
Yeah language models are cool but did you realize that CSS 3D environments is now ten-year-old technology? Ten. Years.
( by
@keithclarkcouk
)
1
0
3
@_LeyVerse
Yeah, I was thinking about that, that is the next challenge. I do plan to release the code
1
0
3
@imaurer
@simonw
@ggerganov
@andreweduffy
@samuel_colvin
@jxnlco
xaedes is working on function calling for the OAI-compatible server in llama.cpp, so that'll be turnkey and potentially drop-in for those Pydantic use cases
0
0
3
There's a beauty and a mystery to building on base models like Llama - rather than following instructions, you're setting it up to simulate the output of some imagined process (human or machine) and working backwards from that.
0
0
3
That stage of a codebase's lifecycle where none of the names in the code match their names in the UI
1
0
3
With constraints on sampling and a quite minimal prompt llama-2-13b has proven more than capable of generating a plausible game world. It only goes off the rails like 40% of the time now
0
0
3
@StayuKasabov
Yes, I intend to do that. Well, the objects came out of
@tripoai
TripoSR so they have a colored mesh
2
0
3
@ReedSealFoss
I plan to put it up soon, I think it's in a good spot now for an initial release. The models are all local, but eventually I envision having the option to run remote as well. The models as well as most of the world state are coordinated by a little Python server.
1
0
2
@pa_schembri
@Teknium1
Yeah! This approach is model-agnostic so I'm interested in experimenting with different finetunes when the engine is more mature.
0
0
2
@mervenoyann
@huggingface
I'll look into it! Does that require the source code to be released/accessible?
0
0
3
I think I figured out the math to do reverse projection and go from screen coordinates to coordinates on a 3D surface. The projection matrix produces 4D vectors, homogenous coordinates, so its inverse isn't directly applicable as you only have 2D screen position. BUT you know
Finally figured out how to compose a transform matrix for a surface in a CSS 3D scene and translate screen coordinates to element coordinates (more in thread)
1
0
5
0
0
3
@MIncarnator
This is three.js in a desktop browser but I do have VR ambitions. For backend, I don't know much about ComfyUI but I'm trying to build this in a way where the model capabilities/workflows can be swapped out.
0
0
2
Did a bunch of reworking on my llama.cpp text adventure, back to something not half bad! And places now have exits. Blue is user input, green is model output (Llama 2 13B)
0
0
2
Wondering if emotion is a necessary part of effective or performant memory. Maybe there's something to
@SocialAGI
0
0
2
Used Stable Diffusion inpainting to generate this year's gift tags. Settled on "four different christmas gift tags highly detailed cozy painting", 60 iterations
1
0
2
@gary_doesnt_lai
@StabilityAI
Sure! The code is available but there are a few steps involved in its current state:
- the code for this demo is on a branch on my fork of zero123:
- it can be installed in a virtualenv with `pip install 'zero123-inference @ git+'`
1
0
2
@tripoai
@StayuKasabov
Thanks! The API looks awesome and I'm sure would yield even better results. I love that we have the choice to run local or remote.
0
0
2
My question about the Rockstar/GTA thing is why are folks even using scanf?
0
0
2
The reverse, screen to element coords, wasn't obvious. I could invert the matrix but the w and z values seemed to be necessary to reverse the transformation. What I learned is that a 2D projected point corresponds to a ray in 3D space, and I specifically want that ray at z=0.
1
0
2
@alonsosilva
@jxnlco
@andinfinity_eu
@Teknium1
@AskMarvinAI
llama.cpp is designed for performant inference on consumer grade hardware, typically for running locally
1
0
2