Nelson Chen's Blog

Bambu Lab Store Filament Tracker

Wed, 11 Feb 2026 20:10:00 -0800

I’ve been running a Bambu Lab Store filament tracker at bbltracker.com.

I recently got into 3D printing. A lot of what I print is practical: last-millimeter adapters, things like robot vacuum ramps around the house, and other tiny fixes that make daily life less annoying.

Bambu Lab released the P2S, their flagship midrange printer, and I picked one up after hearing how polished their ecosystem is. People call them the Apple of 3D printers, and I can see why. The hardware is solid, but the software and ecosystem are really what make it shine.

There is also a bit of a format war around filament. Bambu sells filament with RFID tags that are not easy for third parties to mimic. The RFID flow makes loading filament easy and automatically applies Bambu-tuned profiles. Some people feel those profiles run too fast, but for my functional prints, they have worked great.

Where the frustration started

I like having color options, and color can be functional too. But stock has been rough for a while, even after the holiday season.

That rough stock situation (and the need for a tracker) is really a symptom of how popular Bambu has gotten, especially over the holidays:

Source: Reddit comment

Then there is the constant bulk-sale mechanic on the Bambu Lab Store: buy 4+, get a strong discount, buy more, save more. In practice, that falls apart when the combinations you want are never all in stock at the same time.

At the time, the store UI also made this harder than it needed to be. Out-of-stock options were not clearly crossed out, so building a cart felt like guesswork.

That made me angry, but in a constructive way. So I built a tracker.

What the tracker does

At a high level, the tracker monitors store inventory over time, keeps historical snapshots, and renders a dashboard so I can quickly answer:

What is actually in stock right now?
What appears stable versus constantly draining?
What was the last major restock spike?
Are there ETA hints for restocks?
Is this likely to go out of stock soon?

I started with only the US store, then expanded to EU, CA, UK, AU, and Asia coverage.

I also reused a stock-quantity verification trick I had learned while documenting openpilot’s Toyota security key journey.

Building it fast (and keeping it moving)

I used a lot of Google Antigravity while building this. Web scraping is tedious work, and large language models helped a lot with iteration speed and the constant tweaks.

The codebase is not pretty in every corner, but LLMs have made it very workable to maintain and improve continuously.

Later, I added depletion-rate estimation so the tracker can make a rough guess about when something might go out of stock.

Things that happened after launch

After I released it and posted it on /r/BambuLab, a bunch of interesting things happened:

Bambu nerfed the original stock-count hole the tracker used.
For a while, counts were capped at 200 in the US view, which reduced visibility.
Later, that cap moved to 400 in the US store. Maybe my tracker’s utility was actually a factor in that decision? I do see traffic from China, where there is no Bambu Lab Store of the same codebase as far as I can see.
Bambu improved their UI and added clearer out-of-stock cross-outs.
I added more quality-of-life indicators, including “stable stock,” ETA context, and restock-spike visibility.
I added a GoFundMe and a nice patron funded a custom domain and hosting for the tracker.

Backend at a glance

Keeping this online reliably has mostly been a deployment and operations problem:

Google Cloud Run is heavily used for scheduled jobs, scaling, and reliability.
Cloudflare Pages hosts the output and leaves room for future expansion.

If you print a lot and buy filament in batches, this has been much better than manually refreshing product pages and hoping for the best.

Surviving a power outage with fiber ONT and PoE setup (Frontier FOX222)

Sun, 06 Jul 2025 18:12:00 -0800

For some reason, it’s been a bit hard to find information on how to setup my particular fiber ONT so that it can survive a power outage with a PoE setup.

While of course, one could power a whole house with solar and batteries, I wanted to find a way to keep my fiber ONT and router powered during a power outage without having to invest in a full home battery system.

Costco recently had on sale a small EcoFlow RIVER 3 Plus Portable Power Station, which I installed into my home network setup with the power station being next to my router.

It’s been wonderful as an UPS!

My setup is a bit peculiar as my Frontier FOX222 ONT is a bit far from the power station, but the power station is next to the router and there is an Ethernet cable running from the ONT to the router. The RIVER 3 is also next to the router.

The only way to provide power to the ONT without also having to plumb AC backed by the power station: A PoE injector on the Ethernet cable between the ONT and the router and a splitter at the ONT end to convert the PoE to 16V DC power for the ONT.

The FOX222 ONT is powered by a 16V DC power supply. There is an “Auxiliary Power Supply” port on the side of the power supply. Unfortunately, there is no printed information on what acceptable inputs are acceptable for this port. Nor is there documentation on the power supply itself on acceptable voltages. And size, that took forever and was inconsistent.

Information is also a bit scattered and/or some people have done permanent modications. Some sources have even said 12V is all you need but this is wrong! The information may be applicable to a different ONT model.

Through some trial and error and quite some Amazon returns, I’ve have an Amazon checklist of the requirements for what is needed.

My environment:

Frontier FOX222 Fiber ONT (Optical Network Terminal)
Cyberpower CA25U16V2 Power Supply

Requirements:

You need a PoE injector that can provide at least 30W. The power supply itself at the end of the chain is rated for 16V at 1.6A but it is unlikely to use the full 1.6A.
- TRENDnet Gigabit Power Over Ethernet Plus Injector, PoE+ (30W)
You need a PoE splitter that can provide 16V DC output or something close to it. The FOX222 ONT requires 16V DC input. It will not run with 12V DC input. The 18V DC output is within range and works.
- LINOVISION 30W Gigabit PoE Splitter to DC 5/9/12/18V Output, Wide Voltage Input
The power supply connector has a DIN-like connector on one end. It must be modified so that when there is no AC power, the ONT will still power on and provide data.
- The modification is easily reversible with a small piece of alumnium foil.
- Image from https://www.instructables.com/I-211M-L-ONT-Enable-Data-While-on-Battery-Power/
- If not modified, apparently the ONT will refuse to power on the data port. I did not care to try.
The “Auxiliary plug to the the power supply is a 4.0mm x 1.7mm barrel connector. I don’t know why some some sources say it’s a 4.5mm x 3.0mm barrel connector. Or have other bizzare sizes such as “1.3mm x 3.5mm”.
- 4.0mm x 1.7mm barrel connector
You’re adding some Ethernet cable segments. Some small thin cables is fine.
- Monoprice’s SlimRun cables seem to be excellent. I have a few 1 ft patch cables

Installation:

Plug the PoE injector into the wall and connect it to the router on the WAN side. Use the short Ethernet cables to add small segments as needed.
Connect the PoE splitter to the ONT on the other end of the Ethernet cable. Use the short Ethernet cables to add small segments as needed.
The LINOVISION PoE splitter has a 5.5mm x 2.1mm barrel connector male as an output. Connect it to the PoE adapter’s cable with the 4.0mm x 1.7mm barrel connector.
Plug the 4.0mm x 1.7mm barrel connector into the “Auxiliary Power Supply” port on the Cyberpower CA25U16V2 power supply.
Take out the wire between the ONT and the power supply and modify the plug as shown in the image above. This is done by stuffing in a small piece of aluminum foil into the barrel connector. It apparently shorts some pins together to allow the ONT to power on and provide data when there is no AC power.
Unplug the AC cord from the Cyberpower CA25U16V2 power supply.
Plug in the DC barrel connector into the power supply’s “Auxiliary Power Supply” port.
It should power on!
Do not plug in the AC cord! as if you plug it in, the ONT will reboot. This is actually bad. If the AC power goes out, the ONT will reboot. Just leave it powered by the DC power supply exclusively and from the PoE injector.

Setup Image:

Operation:

Leave the AC cord from the power supply unplugged. If you plug it in, power outages will disrupt service. You’re welcome to test this yourself once you have the setup working completely on DC power.

Things I don’t know:

Does this work the with the FRX523? It’s apparently some sort of successor to the FOX222.
Will there by long term issues with the ONT being powered by a 18V DC power supply?

Comments:

Questions and suggestions are welcome to https://bsky.app/profile/mindflakes.com .

References:

https://www.dmdtech.org/2024/11/30/power-your-frontier-and-others-ont-w-poe/
- Gave a hint that 16V is an absolute requirement
- Provided a hint to the LINOVision PoE splitter’s 18V output being acceptable
https://www.instructables.com/I-211M-L-ONT-Enable-Data-While-on-Battery-Power/
- Provided the image above
- Implies that the power supply is happy to take a 12V DC input but this is wrong for the FOX222 ONT, at least.
- Plug size is wrong, it’s 4.0mm x 1.7mm!
- Only thing right seems to be the modification of the power supply plug with aluminum foil

Talky Pet Watcher, a cat watching AI

Sat, 01 Feb 2025 06:08:55 +0000

Github Project: talky_pet_watcher

I’m allergic to cats 😢. But a friend asked me to watch their cat. My family was more than willing to help but to be safe, I wanted to also monitor the cat to make sure she didn’t get into any shenanigans that were too much.

With some cheap TP-Link Tapo cameras, I set up a system to watch the cat remotely and ensure she was safe while I could not be next to her. It’s a Telegram bot that also sends me a message whenever the cat is doing something interesting.

It watches multiple cameras and tries to aggregate the data to provide insights on the cat’s behavior.

Here’s the project description:

Talky Pet Watcher is a tool to watch a series of ONVIF webcams and their streams. It captures video clips, uploads them to Google AI for processing, and uses a generative model to create captions and identify relevant clips. The results are then delivered to a Telegram channel. This project is designed to help pet owners keep an eye on their pets and share interesting moments with friends.

This was slapped together in a few days, as I only watched the cat for two weeks.

I was also interested in Google Gemini and how cheap they claimed to be for analyzing video. I figured that this would be a good way to test it out.

What went well:

The system was able to capture interesting moments effectively.
It was able to concantate multiple camera’s POV to provide interesting views.
The cameras were cheap!
Putting the results on Telegram was a good way to share the results with the cat’s owner and friends.

What did not go well (and there are many!):

I did not implement history, so the system basically reported the same thing over and over again.
Getting reliable clips from multiple cameras when motion was detected was iffy due to the TP-Link Tapo camera’s iffy web servers. It would stall and halt a lot.
The Google Gemini AI could only analyze video clips at 1FPS. For an agile kitten, this can sometimes make wrong assumptions about what the cat is doing.
Implementation wise, bun was very crashy and I had to restart it a lot.
The connection to the camera also had to be restarted a lot.

If I had to do it again:

Use something that can pull from a NVR by relative timestamps. There was some Rust NVR but it was too complicated for me to set up in the small amount of time I had.
Implement a history system and maybe some memory bank system.
And so so much more! 😅

comma three faux touch keyboard

Fri, 13 Dec 2024 22:02:00 -0800

🦾 comma three Faux-Touch keyboard

Long arms for those of us with short arms from birth or those who can’t afford arm extension surgery!

Built a faux-touch keyboard for the comma three using a CH552G microcontroller. The keyboard is off the shelf and reprogrammed to emulat a touchscreen.

It’s a bit of a hack but it certainly beats gorilla arm syndrome while driving.

For $5 of hardware, it’s very hard to beat.

The journey started in January 2024 and ended in May 2024. Though it, I had to struggle with:

CH552G programming
Linux USB HID Touchscreen protocol
Instructions and documentation

It’s also been through that period of unintended usage. The Frogpilot project has had users who were surprisingly interested in using it as an alternative for some GM vehicles where there is no ACC button. They use the keyboard to touch the screen where ACC would be.

Part of the work also meant upstreaming to the the CH552G community. While I doubt it’ll have any users, it’s nice to give back to the community. There is now a touchscreen library for the CH552G.

The project is open source and can be found at:

https://github.com/nelsonjchen/c3-faux-touch-keyboard

There’s a nice readme that explains how to build and flash the firmware.

comma three faux touch keyboard

Fri, 13 Dec 2024 00:00:00 +0000

🦾 comma three Faux-Touch keyboard

Long arms for those of us with short arms from birth or those who can’t afford arm extension surgery!

Built a faux-touch keyboard for the comma three using a CH552G microcontroller. The keyboard is off the shelf and reprogrammed to emulat a touchscreen.

It’s a bit of a hack but it certainly beats gorilla arm syndrome while driving.

For $5 of hardware, it’s very hard to beat.

The journey started in January 2024 and ended in May 2024. Though it, I had to struggle with:

CH552G programming
Linux USB HID Touchscreen protocol
Instructions and documentation

The project is open source and can be found at:

https://github.com/nelsonjchen/c3-faux-touch-keyboard

There’s a nice readme that explains how to build and flash the firmware.

Replicate.com openpilot Replay Clipper

Thu, 12 Oct 2023 22:28:00 -0800

The replay clipper has been ported to Replicate.com!

https://replicate.com/nelsonjchen/op-replay-clipper

Along with it comes a slew of upgrades and updates:

GPU accelerated decoding, rendering, and encoding. NVIDIA GPUs are provided to the Replicate environment and greatly speed up the clipper.
Rapid fast downloading of clips. Instead of relying on replay to handle downloads sequentially in a non-parallel manner, we use the parfive library to download underlying data in parallel.
Comma Connect URL Input. No need to mentally calculate the starting time and length. Just copy and paste the URL from Comma Connect.
Video/UI-less mode. Don’t want UI? You can have it.
360 mode. Render 360 clips
Forward Upon Wide. Render clips with the forward clip upon the wide clip
Richer error messages to help pinpoint issues.
No more having to manage GitHub Codespaces. Replicate handles all the setup and cleanup for you.

Unfortunately, there is a cost. It’s a very small cost but technically Replicate.com is not free. Expect to drop a cent per render. Thankfully, you have a lot a trial credits to burn through and the clipper can run on a free-ish tier.

Colorized-Project: A Project-Color restoration.

Wed, 01 Feb 2023 22:28:00 -0800

GitHub Link: https://github.com/nelsonjchen/Colorized-Project

On IntelliJ IDEs, I’ve been using Project-Color to set a color for each project. This is useful for me because I have a lot of projects open at once, and it’s nice to be able to quickly identify which project is which.

Unfortunately, Project-Color hasn’t been updated in a while, and it doesn’t work with the latest versions of IntelliJ.

After a few months of using IntelliJ without Project-Color, I decided to try to fix it. I’ve released the result as Colorized-Project.

Thre are a few rough edges, but it’s mostly functional. I’ve been using it for a few weeks now, and it’s been working well. I’ve also submitted it to the JetBrains plugin repository as well.

It still needs some polish like a nicer icon, and I’d like to add a few more features.

Colorized-Project

Wed, 01 Feb 2023 00:00:00 +0000

GitHub Link: https://github.com/nelsonjchen/Colorized-Project

Remember your projects by color! Colorize the window somehow to differentiate your open projects. Updated and maintained fork of Project-Color.

Working on multiple projects simultaneously with Jetbrains products based on IntelliJ can be difficult, since they all look the same at a glance. This plugin solves that by giving the user the option to colorize each project’s window in a noticeable way.

GitHub Wiki SEE: Search Engine Enablement: Year One

Thu, 13 Oct 2022 21:04:00 -0800

It’s been near a year since I supercharged GitHub Wiki SEE with dynamically generating sitemaps.

Since then a few things have changed:

GitHub has started to permit a select set of wikis to be indexed.
They have not budged on letting un-publically editable wikis be indexed.
There is now a star requirement of about 500, and it appears to be steadily decreasing.
Many projects have reacted by moving or configuring their wikis to indexable platforms.

As a result, traffic to GitHub Wiki SEE has dropped off dramatically.

This is a good thing, as it means that GitHub is moving in the right direction.

I’m still going to keep the service up, as it’s still useful for wikis that are not yet indexed and there are still about 400,000 wikis that aren’t indexed.

Hopefully GitHub will continue to move in the right direction and allow all wikis to be indexed.

openpilot Replay Clipper

Wed, 12 Oct 2022 19:42:00 -0800

End to End Longitudinal Control is currently an “extremely alpha” feature in openpilot that is the gateway to future functionality such as stopping for traffic lights and stop signs.

Problem is, it’s hard to describe its current deficiencies without a video.

So I made a tool to help make it easier to share clips of this functionality with a view into what openpilot is seeing, and thinking.

GitHub repository

It’s a bit heavy in resource use though. I was thinking of making it into a web service but I simply do not have enough time. So I made instructions for others to run it on services like DigitalOcean, where it is cheap.

It is composed of a shell script and a Docker setup that fires up a bunch of processes and then kills them all when done.

Hopefully this leads to more interesting clips being shared, and more feedback on the models that comma.ai can use.

I later also ported this to Replicate here.

Datasette-lite sqlite-httpvfs experiment POC with the California Unclaimed Property database

Tue, 11 Oct 2022 20:36:00 -0800

There’s a web browser version of datasette called datasette-lite which runs on Python ported to WASM with Pyodide which can load SQLite databases. I grafted the enhanced lazyFile implementation from emscripten and then from this implementation to datasette-lite relatively recently as a curious test. Threw in a 18GB CSV from CA’s unclaimed property records here

https://www.sco.ca.gov/upd_download_property_records.html

into a FTS5 Sqlite Database which came out to about 28GB after processing:

POC, non-merging Log/Draft PR for the hack:

https://github.com/simonw/datasette-lite/pull/49

You can run queries through to datasette-lite if you URL hack into it and just get to the query dialog, browsing is kind of a dud at the moment since datasette runs a count(*) which downloads everything.

Elon Musk’s CA Unclaimed Property

Still, not bad for a $0.42/mo hostable cached CDN’d read-only database. It’s on Cloudflare R2, so there’s no BW costs.

Celebrity gawking is one thing, but the real, useful thing that this can do is search by address. If you aren’t sure of the names, such as people having multiple names or nicknames, you can search by address and get a list of properties at a location. This is one thing that the California Unclaimed Property site can’t do.

I am thinking of making this more proper when R2 introduces lifecycle rules to delete old dumps. I could automate the process of dumping with GitHub Actions but I would like R2 to handle cleanup.

openpilot Replay Cliper

Sat, 01 Oct 2022 00:00:00 +0000

Replicate: https://replicate.com/nelsonjchen/op-replay-clipper

https://github.com/nelsonjchen/op-replay-clipper/

📽 Capture short ~30 second route clips with the openpilot UI included, and the route, seconds marker branded into it.

Designed to try to encourage more interesting clips to be shared, especially with newer functionality such as “end to end longitudinal control”.

Originally a bit heavy, so instructions were made for others to run it on services like DigitalOcean/GitHub Codespaces. Since then, the clipper has been ported to Replicate which provides GPUs and a nice WebUI.

The UI rendering is composed of a shell script and a Docker setup that fires up a bunch of processes and then kills them all when done. On Replicate, it is wrapped by a Python script that handles the UI and non-UI processing of video and downloads.

Released Gargantuan Takeout Rocket

Sat, 16 Apr 2022 21:04:00 -0800

Finally released GTR or Gargantuan Takeout Rocket.

GitHub repository

Gargantuan Takeout Rocket (GTR) is a toolkit of guides and software to help you take out your data from Google Takeout and put it somewhere else safe easily, periodically, and fast to make it easy to do the right thing of backing up your Google account and related services such as your YouTube account or Google Photos periodically.

It took a lot of time to research and to workaround the issues I found in Azure.

I also needed to take apart browser extensions and implement my own browser extension to facilitate the transfers.

Cloudflare Workers was also used to work around issues with Azure’s API.

All this combined, I was able to takeout 1.25TB of data in 3 minutes.

Now I’m showing it around on Twitter, Discord, Reddit, and more to users who have used VPSes to backup Google Takeout or have expressed dismay at the lack of options for users who are looking to store archives on the cheap. The response from people who’ve opted to stay on Google has been good!

There is also a project page with additional details here.

Gargantuan Takeout Rocket

Sat, 16 Apr 2022 00:00:00 +0000

https://github.com/nelsonjchen/gtr

Used to do this with a VPS. It was still too slow to backup 1.25TB. TOOK HOURS! Built a Rocket.

It is comprised of a two major components:

A “GTR Proxy” Cloudflare Worker that HTTP3-izes the the Azure API and is capable of proxying base64 URLs to Google Takeout URLs.
A browser extension that coordinates all this and constructs a job plan to execute.

These two components are infinitely scaleable and can be used to backup Google Takeout at breathtaking, unprecedented speeds.

The GTR repo contains a guide on how to setup all of this.

A future revision or successor may target S3 or S3-like APIs. In particular, can we target Cloudflare’s R2 when it comes out?

Cell Shield

Wed, 28 Jul 2021 18:47:00 -0800

I built https://cellshield.info to scratch an itch. Why can’t I embed a spreadsheet cell?

You can click on that badge and see the sheet that is the data backing the badge.

It is useful for embedding a cell from an informal database that is a Google Spreadsheet atop of some README, wiki, forum post, or web page. Spreadsheets are still how many projects are managed and this can be used to embed a small preview of important or attention grabbing data.

It is a simple Go server that uses Google’s API to output JSON that https://shields.io can read. Mix it in with some simple VueJS generator front-end to take public spreadsheet URLS and tada.

I picked to base my implementation on shields.io’s as they make really pretty badges and have a lot of nice documentation and options.

I run it on Cloud Run to keep the costs low and availability high. It also utilizes Google’s implementation of authorization to get access to the Sheets API.

The badge generator web UI can also make BBCode for embedding (along with modern Markdown and so on).

I’ve already used it for this and that. The former outputs a running bounty total and the latter is community annotation progress.

A major Game Boy development contest is using it labels to display the prize pool amount: https://itch.io/jam/gbcompo21

Why I built this?

I built it for the Comma10K annotation project. I contributed the original badge which attempted to use a script that was used in the repo to analyze the completion progress. Unfortunately, this kept on diverging from the actual progress because the human elements kept changing the structure and organization and I had the badge removed. The only truth is what was on the spreadsheet.

Popularity

Compared to my other projects, I don’t think the popularity will be too good for this project though. I think it might have a chance to spread via word of mouth but it’s not everyday that a large public collaborative project is born with a Google Spreadsheet managing it. The name isn’t that great either as it probably conflicts with anti-wireless hysteria. Also, it’s not like there’s an agreed upon community of Google Spreadsheet users to share this knowledge with.

I also shamelessly plugged it here but I think the question asker might be deceased:

https://stackoverflow.com/questions/57962813/how-to-embed-a-single-google-sheets-cells-text-content-into-a-web-page/68505801#68505801

At least it was viewed 2,000 times over the last two years.

Maintenance

It’s on Cloud Run. The cost to keep it running is low as it sometimes isn’t even running and the Sheets V4 API it uses was recently declared “enterprise-stable”. Because of this, it will be super easy to maintain and cheap to run. I don’t expect any more than $2 a month.

GitHub Wiki SEE: Search Engine Enablement

Tue, 27 Jul 2021 18:12:00 -0800

I built this in March, but I figure I’ll write about this project better late than never. I supercharged it in June out of curiosity to see how it would perform. It’s just a ramble. I am not a good writer but I just wanted to write stuff down.

TLDR: I mirrored all of GitHub’s wiki with my own service to get the content on Google and other search engines.

If you’ve used a search engine to search up something that could appear on GitHub, you are using my service.

https://github-wiki-see.page/

Did you know that GitHub wikis aren’t search engine optimized? In fact, GitHub excluded their own wiki pages from the search results with their robots.txt. “robots.txt” is the mechanism that sites can use to indicate to search engines whether or not to index certain sections of the site. If you search on Google or Bing, you won’t find any results unless your search query terms are directly inside the URL.

The situation with search engines is currently this. For example, if you wanted to search for information relating to Nissan Leaf support of the openpilot self-driving car project in their wiki, you could search for “nissan leaf openpilot wiki”. Unfortunately, the search results would be empty and would contain no results. If you searched for “nissan openpilot wiki”, “https://github.com/commaai/openpilot/wiki/Nissan" would show up correctly because it has all the terms in it. The content of the GitHub wiki page is not used for returning good results.

It has been like this for at least 9 years. GitHub responded that they have declined to remove it from robots.txt as they believe it can be used for SEO abuse. I find this quite unbelievable. The mechanism to prevent this is to utilize rel="nofollow" on all user generated <a> links in the content and with newer guidance suggesting rel="nofollow ugc". By removing the content from the index, they have blinded a large portion of the web’s users to the content in GitHub wiki pages.

Until I made a pull request to add a notice to their documentation that wikis are not search engine optimized, there was absolutely no official documentation or notice about this decision. I still believe it is not enough as nothing in the main GitHub interface notes this limitation.

I believe there are many dedicated users and communities that are not aware of this issue who are diligently contributing to GitHub wikis. This really frustrated me. I was one of them. Many others and I contributed a lot to the [comma.ai openpilot wiki][op_wiki] and it was maddening that their work and my work were not indexed.

I noticed that a lot of technical questions had Stack Overflow mirrors rank highly and yet they weren’t Stack Overflow.

So what I did was this: I made a proxy site without a robots.txt that would mirror GitHub wiki content to a site that did not have a restrictive robots.txt. I put it up at https://github-wiki-see.rs. I called it GitHub Wiki SEE for GitHub Wiki Search Engine Enablement. It’s kind of a twisted wording of SEO as I see it as enabling and not even “optimizing”.

What’s the technology behind it?

The current iteration is a Rust Actix Web application. It parses the URL and makes requests out to GitHub to get the content. It then scrapes the content for relevant HTML and reformats it to be more easily crawl-able. It then serves the content to the search engine and users who come in from a search engine with a link to the project page and reasoning at the bottom as a static pinned element. Users aren’t meant to read that page and are only meant to browse to the original GitHub page.

With these choices, I could meet these requirements:

Hosted it on Google Cloud Run just in case it doesn’t become popular.
Made it easy to deploy.
Made it cheap.
Made it small to run on Cloud Run.
Made it low maintenance.

Another considering was that it needed to not get rate limited by GitHub. To do this, I made the server return an error and quit itself on Cloud Run if it detected a rate limiting response to cycle to a new server with a new client IP address. Unfortunately, this limited the number of requests each server could handle at a time to a minimum of one.

Supercharging it

I originally built the project in March. I monitored the Google Search Console like a hawk to see how the openpilot stuff I had contributed to was responding to the proxy setup. Admittedly, it got a few users to click through here and there and I was pretty happy. For a month or two, I was pretty satisfied. However, I kept on thinking about the rest of the users who lamented this situation. While some did take my offer to post a link somewhere on their blog, README, or tweet, the adoption wasn’t as much as I had hoped.

I had learned about GH Archive which is a project that archived every event of GitHub since about 2012. I also learned that the project also mirrors its archive on BigQuery. BigQuery allows you to search large datasets with SQL. Using BigQuery, I was able to determine that about 2,500 GitHub Wiki pages get edited every day. In a week, about 17,000. A month? 66,000 pages. My furor at the blocking of indexing increased.

I decided to throw $45 at the issue and run a big mega-query and queried all of GH Archive. Over the lifetime of GH Archive, 4 million pages were edited. I dumped this into a queue and processed it with Google Cloud Functions to run the HEAD query on all the pages in the list and re-exported it to BigQuery. 2 million pages were still there.

You can find the results of this in these public tables on BigQuery:

github-wiki-see:show.checked_touched_wiki_pages_upto_202106
- Checked with HEAD
github-wiki-see:show.touched_wiki_pages_upto_202106

With this checked_touched_wiki_pages_upto_202106 table, I generated 50MB of compressed XML sitemaps for the GitHub Wiki SEE pages. I called them seed sitemaps. The source for this can be found in the sitemap generator repo.

BigQuery is expensive, so I made another sitemap series for continuous updates. An earlier version used BigQuery, but the new version runs a daily GitHub Actions job that downloads the last week of events from GH Archive, processes them for unique Wiki URLs and exports them to a sitemap. This handles any ongoing GitHub Wiki changes. The source for this can also be found in the sitemap generator repo.

The results

I am getting thousands of clicks and millions of impressions every day. GitHub Wiki SEE results are never at the top and average a position of about 29 or the third page. The mirrored may not rank high and I don’t care as the original content was never going to rank high anyway so the content appearing at all in the search results of thousands of users even if it was sort of deep satisfies me.

All in all, a pretty fun project. I am really happy with the results.

My contributions to the openpilot wiki are now indexed.
Everyone else’s contributions to their wikis are now indexed as well.
I learned a few things or two about writing a Rust web app.
I learned a lot about SEO
I feel I’ve helped a lot of people who need to find a diamond in the rough and the diamond isn’t in the URL.
I feel a little warm that I have a high traffic site.

I have no intention of putting ads on the site. It costs me about $25 a month to run but for the high traffic it gets, the peace of mind I’ve acquired, and the good feelings of helping everybody searching Google, it is worth it.

I did add decommissioning steps. If it comes to it, I’ll make the site just directly redirect but after GitHub takes Wikis off their robots.txt.

Things I Learned

This is a grab-bag of stuff I learned along the way.

Sitemaps are a great way to get search engine traffic to your site.
Large, multi-million page sitemaps can take many months to process. Consider yourself lucky if it parses 50,000 URLs a day.
- The Google Search Console saying “failed to retrieve” is a lie. It means to say “processing” but this is apparently a long standing stupid bug.
GoogleBot is much more aggressive than Bing.
- Google’s Index of my proxy site is probably a lot larger than Bing’s by a magnitude or two. For every 30, 40, or even 100 GoogleBot requests, I get one Bing request.
- This is important for serving DuckDuckGo users as it is their index.
The Rust Tokio situation is a bit of a pain as I am stuck on a version of Tokio that is needed for stable Actix-Web. As a result, I cannot use modern Tokio stuff that modern Rust crates or libraries use. I may port it to Rocket or something in the future.
A lot of people search for certification answers and test answers on GitHub Wikis.
GitHub Wikis and GitHub are no exception to Rule 34.
Cloud Run is great for getting many IP addresses to use and to swap out if any die. The outgoing IP address is extremely ephemeral and changing it is a simple matter of simply exiting if a rate limit is detected.
- This might not always work. Just exit again if so.
GCP’s UI is nicer for manual use compared to AWS.
The built-in logging capabilities and sink of GCP products are also really nice. I can easily re-export and/or stream logs to BigQuery or to a Pub Sub topic for later analysis. I can also just use the viewer on the UI to see what is going on.
I get fairly free monitoring and graphing from the GCP ecosystem. The default graphs are very useful.
Rust web apps are really expensive to compile on Google Cloud Build. I had to use 32-core machines to get it to compile in underneath the default timeout of 10 minutes. I eventually sped it up and made it cheaper to build by building on a pre-built image along with a smaller instance but it is still quite expensive.
The traffic from the aggressive GoogleBot is impressively large. Thankfully, I do not have to pay the bandwidth costs.
My sloppy use of the HTML Parser may be causing errors but RAM is fairly generous. It’s Rust too so I don’t really have to worry that much about garbage collection or anything similar.
Google actually follows links it sees in the pages. When I started, it did not. Maybe it does it only on more popular pages.
The Rust code is really sloppy, but it still works.

GitHub Wiki SEE

Thu, 01 Apr 2021 00:00:00 +0000

https://github-wiki-see.page/

A project to get GitHub Wikis indexed by Google Search.

Explanation is right on the front page.

If a search engine can’t see it, it may as well be invisible.

Long-term project involving Rust, BigQuery, Cloudflare Workers, and lots of random hosting.

Probably one more the more visited sites on the internet and a continous effort to keep costs low.

Has caused GitHub to revisit their policy on a limited subset of Wikis meeting some criteria.

Will continue until all Wikis are indexed even if they are publically editable or don’t meet some star count criteria.

Personally, I just got really dismayed all this documentation I was doing on a Wiki wasn’t visible to Google. Then I realized it was a good idea to make it visible. Then I realized others don’t realize it was invisible at all. This was a problem.

The short lived life of Breakout Bots for Zoom Meetings

Sat, 12 Sep 2020 19:42:00 -0800

With the pandemic happening, it’s been tough for many organizations to adapt. We’re all supposed to be together! One way organizations have been staying together is by using Zoom Meetings.

At work, we’ve been using a knock-off reseller version called RingCentral Meetings. From looking at their competitors, Zoom has pulled out all the stops to make their meeting experience the most efficient, resiliant, rather cheapest, and reliable experience.

One of these features is “Breakout Rooms”. With breakout rooms, users can subdivide their meetings to make mini-meetings from a bigger meeting. For most of the year, there has been an odd restriction on Breakout Meetings though.

You cannot go to another breakout room as an attendee without the host reassigning you unless you are a Host or a Co-Host.

Obviously, this can result in a lot of load upon the poor user who is desginated the host. Even the Co-Hosts can’t assign users to another breakout room.

So I made a bot:

https://github.com/nelsonjchen/BreakoutRoomsBotForZoomMeetings

It’s quite a hack, but it basically controls a web client that is a Host and assigns users based on chat commands and on attendees renaming themselves.

Last night, I just finished finally optimizing the bot. It should be able to handle hundreds of users and piles of chat commands.

This morning I learned that Zoom will add native support for switching:

https://www.reddit.com/r/Zoom/comments/irkm82/selfselect_breakout_room/

It was fun, but as brief as the bot existed, the main purposes are numbered. Maybe someone can reuse the code to make other Zoom Meeting add-on/bots?

Technical and Learnings

I knew about this issue for many months but I never made a bot because there wasn’t an API. I waited until I was very disatisified with the situation and then made the bot. I should have just done it, without the API.
Thankfully, by doing this quickly, I only really spent like a week of time on this total. With some of the tooling I used, I didn’t have to spend too long debugging the issues.
The original implementation of this tool was a copy and paste into Chrome’s Console. This is a much worse experience for users than a Chrome extension. Kyle Husmann contributed a fork that extensionized the bot and it is totally something I’ll be stealing for future hacks like this.
Zoom got to where they are by doing things their competitors did not and rather quickly. This feature/feedback response is great and is why they’ve been crushing. I feel this is a feature that would have taken their competitors possibly a year to implement. At this time, Microsoft Teams is also looking to implement Breakout Rooms.
Zoom’s web client uses React+Redux. They also use the Redux-Thunk middleware. They do stick some side-effects in places where things shouldn’t but whatever, it works. It did make integration very hard if not impossible at this layer for my bot. However, subscribing to state changes was and is very reliable.
RxJS is a nice implementation of the Reactive Pattern with Observables and stuff like with RX.NET. Without this, I could not fit the bot in so few lines of code with the reliability and usability that it currently has.
With RxJS and Redux’s store, I was able to subscribe to internal Redux state changes and transform them into streams of events. For example, the bot transform the streams of renames and chat message requests into a common structure that it then merges to be processed by later operators. Lots of code is shared and I’m able to rearrange and transform at will.
The local override functionality is great if you want to inject Redux Web Tools into an existing minified application. If you beautify the code, you can inject dev tools too. Here’s the line of code I replaced to inject Redux Dev Tools into Zoom’s web client.

Previous:
```
, s = Object(r.compose)(Object(r.applyMiddleware)(i.a));
```
After:
```
, s = window.__REDUX_DEVTOOLS_EXTENSION_COMPOSE__(Object(r.applyMiddleware)(i.a));
```
This was super helpful in seeing how the Redux state changes when performing actions in a nice GUI.
fuzzysort is great! I’m quite surprised at how unsurprising the results can be when looking up rooms by a partial or cut up name.
The underlying websocket connection is available globally as commandSocket. If you observe the commands, they are just JSON and you can inject your own commands to control the meeting programatically.
Since Breakout Room status is sent back via WebSocket, the UI will automatically update. I did not have to go through the Redux store to update the Breakout Room UI.
UI Automation of Breakout Room UI: ~100ms vs webSocket command: 2ms. Wow!
Chat messages are encrypted and decrypted client-side before going out on the Websocket interface. This probably doesn’t stop Zoom from reading the chat messages since they are the ones who gave you the keys. I could have reimplemented the encryption in the bot but after running some replay attacks, the UI did not update. Without the UI updating, I didn’t feel it was safe for the host to not know what the bot sent out on their behalf.
Even if I had implemented the chat message encryption, the existing UI automation of chat messages hovered around 30ms. The profiler also showed that the overhead of automation wasn’t that much and much of the chat message encryption contributed to the 30ms. It wouldn’t have been much of a performance win if I implemented it myself.
Possible Security-ish Issue - Since it isn’t an acceptable bounty issue, and hosts can simply disable chat, I will disclose is possible to DOS the native client by simply spamming chat. If there is too much text, the native clients will lock up and require a restart. Native clients don’t drop old chat logs. Web clients don’t have this issue as they will drop old chat automatically. Last but not least, it is possible for the Host to disable chat in many ways to alleviate such an attack. I wasn’t sure if to let my load_test bot out there or not because of this but after reflection, I think it’s OK.
RxJS is MVP. Or rather the Reactive stuff is MVP. If you can get it into a stream, it’s super powerful. Best yet, you can reuse your experience from other ecosystems. I was previously using Rx.NET for some stuff at work. I’ll be definitely looking at using the Python version of this in the future.
- Though it did seem the differing Reactive stuff have different operators. The baselines are still great though and you can always port operators from one to another implementation.
Rx is really helped if you can use TypeScript. Unfortuntately, I never got to this stage.

That said, this was fun! Now onward to the next thing.

Reverse RDP into Windows on GitHub Actions

Sat, 07 Dec 2019 17:24:55 +0000

Ever wonder what the Desktop of the Windows Runners on GitHub Actions looks like?

Or perhaps you’re missing the ability to RDP into build agents like on Appveyor.

I wrote some steps that use ngrok to make a reverse tunnel possible. I also turned on RDP if it wasn’t on already and set a password.

Take a look here!

https://github.com/nelsonjchen/reverse-rdp-windows-github-actions

Stop Sims 2 Purple Soup or Pink Flashing Crashes on Windows 10 and Modern Hardware with D9VK

Wed, 02 Oct 2019 06:08:55 +0000

Wrote a guide on how to fix the Purple Soup or Pink Flashing Crashes for The Sims 2 on Windows 10 1903+ on powerful gaming computers. It’s great to make this accessible again to gamers with powerful gaming computers.

Guide Link: https://docs.google.com/document/d/19JMr-FQSU3AlF7Kyvcrr7awTiHBDQSUNLG6qfaI6rOs/edit#

General idea: Replace Microsoft DirectX 9 with D9VK.

D9VK is actually actively developed and fixes bugs. It also translates graphics to a stack that is more open and stable.

So this:

Turns to this:

Fast and lightweight headless Qt Installer from Qt Mirrors: aqtinstall

Sun, 02 Jun 2019 22:02:00 -0800

As part of my work on the Barrier project, we needed a way to build it with the [Qt installer]from an online CI/CD service. On some platforms, such as Appveyor, Qt may be preinstalled. But on some other platforms such as Azure Pipelines, Qt may not be pre-installed. Nor on Travis. Nor on CircleCI. For Linux and macOS, there is a saving grace. The Qt libraries can be installed headlessly from their operating system repositories such as apt or brew. But this luxury does not exist as well on Windows. While Chocolatey does offer some Qt packages, they tend to ancient or non-official.

There is a toolkit called CuteCI that let you use the Qt Installer and script the GUI headlessly. It’s janky and big though. Additionally, to pin versions, you must download 3 to 5 GB of Qt Installer to install a pinned version of Qt.

So, I found this: https://lnj.gitlab.io/post/qli-installer/

Apparently it’s possible to mimic the Qt Installer from a Python script.

To save it from possible bit-rot, I forked it to GitHub. I added parallel downloading, Windows support, and a CI system. I pushed Azure-Pipelines and each commit could cause 100 jobs to run. Across all the various platforms. Azure Pipelines did not break a sweat.

CuteCI is still extremely useful if you must use the official installer with official support. But the speeds with my fork were insane. It would normally take many minutes to install Qt with CuteCI. My QLI-Installer fork? 10 or 20 seconds.

But it turns out there’s an even better installer than mine!

https://github.com/miurahr/aqtinstall/

aqtinstall

Classy reusable Python structure
On PyPi. You can do pip install aqtinstall and install aqtinstall for immediate use.

This is also another fork of qli-installer. The CI system was a in a bit of a sorry state though. So I contributed matrix generated jobs to test aqtinstall across Mac, Windows, and Linux. @miurahr was able to dial down the installation and testing madness too and ramp it up in some places. Actual Qt applications are built to test if aqtinstall work. Additionally, there was the insight that mobile platforms generally always require the latest Qt version.

So, if you want a fast and lightweight headless Qt installer that installs from Qt Mirrors and is continually tested, use aqtinstall!

Non-encumbered Windows Bonjour SDK DLLs and Libs

Sat, 01 Jun 2019 19:15:00 -0800

This was one of those adventures where you find a problem and it snowballs from there.

As part of my work on the Barrier project, we needed a way to build it with the Bonjour SDK from an online CI/CD service. Bonjour is used by Barrier to discover other barrier hosts on the same network. On Windows, the SDK files are needed. Other platforms natively come with the library or similar APIs already preinstalled.

But there’s an issue. The SDK is behind a login wall on Apple’s site. Rehosting it might be possible but it might violate some agreement within the installer. Luckily, the sources are open and are under very permissive licenses.

To save some work, I reused the XBMC project’s fork and made my fork of the SDK. I didn’t change any of XBMC’s changes, if any. I added a CI system to build the necessary libraries in the same manner that Apple did.

So, if you need a free, Cloud CI system accessible Apple Windows Bonjour SDK, look at the releases in my repo! The library files are there and users can fork my build to build the stubs their own way as well.

Barrier CI System WIP

Thu, 30 May 2019 22:02:00 -0800

This was one of those adventures where you find a problem and it snowballs from there.

I wanted to use barrier which is a tool for sharing mouses between computers. It is a fork of Synergy, which had gone commercial.

So, I noticed on their GitHub repo that the builds for various platforms was inconsistent. OK, maybe I can help fix their CI systems.

More worringly, there was some Appveyor CI system configured. It was not configured with the appveyor.yml method and the build process was non-transparent. It was also failing.

My current CI system of choice for open source projects is Microsoft’s Azure Pipelines. For open source projects on GitHub, they’ll provide 10 parallel jobs of Mac, Linux, and Windows. Mac and Windows usually only show up as premium options or are severely rate-limited. As far as I can tell, whatever cluster Microsoft has created for their service, it’s really not limited by capacity.

The good news is that they were fans of it and I was able to move them. It’s still a work in progress. The work done here has draw in some general “improve the ecosystem” enhancements which I will elaborate on in future posts.

An Android Accessibility Service for Eventbrite Organizers: PassShout

Fri, 15 Mar 2019 19:15:00 -0800

The Scene:

It’s Saturday. And it’s like so many people have free time on that date.
You’re manning a convention center ticket check-in booth.
Your eyesight just isn’t the best. After looking at hundreds, if not thousands of them, it can be a blur.
You’re using Android.
There’s a long line and people have different pass or ticket types, sometimes even if they are in the same group.
You’re processing thousands of people.
You might not even have a table.
You’re using Eventbrite Organizer.

I have a tool for you! It’s called PassShout.

In addition to a chime, it’ll read out loud the pass/ticket type.

This greatly reduces operational mistakes. PassShout is part of an implementation of a method in occupational safety called Pointing and Calling. PAC has been adopted widely thoughout Asia and the NYC Metro to reduce incidents on trains systems. By calling out the type, things move faster and mistakes are reduced.

Implementation

This is the first Android app I’ve written in Kotlin. It’s amazing what the Android Accessibility API offers for third party programs to make things more accessible. Unfortunately, there does not seem to be an equivalent for iOS.

Kotlin is really easy to read and write. I can see how it’s not Scala and is closer to a usable modern Java. It’s basically the Swift of Android.

NC QEMU

Tue, 25 Dec 2018 00:00:00 +0000

Changes

Azure Pipeline to build x86_x64 target QEMU in MSYS2 on Windows
Upgraded to Capstone with fixes that allow building in MSYS2.

About

“NC QEMU” is a very lightly augmented and experimental fork of QEMU to build on Azure Pipelines and with Windows Hypervisor Platform (WHPX) support for users who care the most about running the x86_64 target fast. This build only cares about the x86_64 target and other targets are not built.

The output is a zip file with some DLLs and qemu-system-x86_64.exe. This QEMU distribution can be run from a folder by itself. Alternatively, qemu-system-x86_64.exe can be dropped into QEMU for Window’s installation directory at C:\Program Files\qemu, replacing the existing version. Some features like USB network redirection might be missing though.

The pipeline provides a differently compiled QEMU compared to QEMU for Windows’s own build instructions which are cross-compiled on Linux. Notably, NC QEMU is built on a Microsoft-provided copy of Windows in MSYS2 with access to the Windows SDK headers for WHPX distributed by Microsoft. The MingW64 Toolkit which QEMU for Windows is built with has unfortunately currently not reproduced the WHPX headers in a free software manner. Users who want a QEMU for Windows with WHPX support but aren’t licensed nor want to accept the terms of installing the Windows 10 SDK are out of luck.

Another benefit is that this project provides a declaratively made and executable pipeline that builds QEMU for Windows. Fork, setup an Azure Pipeline, and adjust if needed. This can be considered a bit “executable documentation” for building a QEMU x86_64 target of this nature. The build logs are public and can be used as reference. For users having trouble building or configuring their systems to build QEMU, this reproducible setup can be quite useful.

Why QEMU+WHPX for `x86_64`

Absolutely not exhaustive:

Acceleration is great. You can reach near-native speeds with acceleration.
WHPX is native to the OS. No foreign kernel drivers/modules/extensions needed.
It’s great for users who don’t have access to Intel’s HAX because they either want Hyper-V and/or AMD support. With AMD CPUs being multi-core bargain monsters, developers and power users on AMD are becoming more numerous.
QEMU+WHPX boots Windows ISOs. HAX currently does not.
QEMU is easier to hack on, script, and developed by many organizations, not one. WHPX support was contributed by Microsoft but is probably most useful for Google’s QEMU-based Android emulators!
Users hacking on or working with QEMU in Windows can bring their work to Linux KVM accelerated systems easier. And vice-versa.

Packer WHPX workaround

The executables provided here do work for Packer. Simply add the directory with qemu-system-x86_64.exe to the PATH. Packer does not currently recognize whpx as a valid argument for the acceleration key. The workaround is to add it to qemu-args as a manual argument.

Additionally, the cpu argument of QEMU does not support cpu=host for whpx. Specify something supported manually.

References

Orbital (PS4 Emulator) Build instructions: https://github.com/AlexAltea/orbital/wiki/Building

Gooey and PyInstaller for easy to write, distribute, and use scripts!

Sat, 15 Dec 2018 20:36:00 -0800

What is:

Easy to write
Easy to distribute and portable
Super Powerful
Cross Platform
Puts computer novices and professionals on the same level?

I think it’s a combination of Gooey and PyInstaller. If you put those two together, you get all those above.

It’s Python, so it’s easy to write. There are so many guides online and a wide assortment of libraries. Most importantly, you get libraries that have a strong eye towards ergonomics. Python is a wonderful beginner’s language that some people even call an executable psuedolanguage.
PyInstaller bundles applications into a single folder or exe so it’s easy to distribute.
You get the full force of Python’s language and its ecosystem behind you.
Python is cross platform, but Gooey’s GUI is also as well. Don’t be expecting to make “Destiny”-style GUIs with it. But do be expecting a surprising amount of native-ness.
While PyInstaller does package the application up into a portable Python script, the existing command line invokation is still left for professionals. Simply add --ignore-gooey to the list of arguments. Now it’s just a plain old script.

The Gooey PyInstaller Demo was presented at a DesertPy meetup in August of 2018. It’s a project showing the development of a simple argument parsing script all the way up to a portable GUI. ~~Future improvements to the demo include some CI scripts and configurations.~~ This was completed on 2018-12-15.

Demo Project Link: https://github.com/nelsonjchen/gooey-pyinstaller-demo

Switched to Hugo

Sun, 29 Nov 2015 00:37:59 -0800

Wow! I think it has been nearly two years since I last updated this blog. I have decided to change the blog a bit and port it over to hugo which is a static site generator written in Go. It is very fast compared to nanoc which I was using. I do not know if I quite agree that this is just as customizable as nanoc but it is clearly faster. I was not taking too much advantage of the previous Foundation based flexibility of the nanoc site and I believe it may had become a burden. Also, bitrot was beginning to seriously set in. Luckily, it is a static site but it does make some tears shed when you have to pin some really old gems and use a really old version of Ruby to build and deploy the site. Unless the kernel syscalls change dramatically, I will probably be able to compile with this version of hugo indefinitely.

Anyway, the nanoc versions troubles me no more. I am using a hugo version of the site now. As an improvement, I have also setup automatic deployment of my master branch to Google’s storage and configured CloudFlare to provide TLS and all the goodies that come with that. I guess I could also start using something like prose.io to write my posts as well. I doubt I will go that far! But, it is very nice to know that is possible.

I am now just using a theme called hyde-y which is off-the-shelf. Of course, I did customize it a little bit. hugo’s overriding system is plain and simple to use.

Why would anyone want summaries on the front page? Just give the complete content.
The titles of the posts on the front page should have the same information.

I addressed all that, and while it was a bit of a hunt, it was fairly clear what needed to be done.

I do not think I will go the full mile with customizing hugo like I did with nanoc. The complexity was hurting more than it helped. I guess I have switched over to being more minimalist since I concieved the original site. Foundation is great if you need the kitchen sink and everything but it is really overkill. It was also another maintenance burden with its rather relentless upgrade schedule. I am taking it slow from here on out.

Running Windows XP using Libvirt inside Ubuntu inside VMWare Fusion

Mon, 06 Jan 2014 18:09:02 -0700

So I’m investigating running Windows XP inside libvirt in Ubuntu on my MacBook for continuous integration testing purposes with Jenkins. I tried for many hours to get it working but Windows would not boot up or it would get stuck on the NTLDR cannot be found issue. As it turns out, I did not enable “Enable hypervisor applications in this virtual machine”. I was under the assumption that I would not need to check that box as I wrongly assumed qemu would handle all the needed machine translation and that an error with bootup is not the fault of lacking the ability to use a CPU feature to emulate something or the disk emulation being borked. With five different ISOs of various pedigrees, I tried installing Windows XP with virt-manager and all of them failed to bootup with various disk image formats like qcow2 or raw. It was only until I checked that checkbox did it then work.

That’s my surprise for today. I really wished I checked that checkbox earlier! It determines if your machine can bootup and only using qemu without kvm will not cut it for booting Windows XP inside libvirt inside Ubuntu inside VMware Fusion.

BeamNG DRIVE on Mac OS X with CrossOver

Sat, 03 Aug 2013 16:06:37 -0700

UPDATE 2015-11-28: Hey BeamNG forum users. These instructions worked for me way back when I was using CrossOver. You might be able to get similar or better results with Wineskin Winery or some other Wine wrapper. And those are free too! I don’t run BeamNG on my local laptop anymore nowadays but I Game on Amazon EC2 and it works fairly well with BeamNG at a very minimal cost.

Gotta make this quick. This post will change.

Here’s how to get BeamNG Drive working in Mac OS X. I’m sure there are equivilent instructions for Wineskin but I have no time to find out.

Make a new bottle in Crossover. Make sure it is a Windows Vista bottle. Run the Wine configuration setting inside it and set the version to Windows 7 in one of the sheets. Enable the Mac Driver so you don’t run it through X11.

Install the Modern DirectX runtime into it. Install BeamNG into it. Mute your speakers. Change the audio options to use OpenAL for output. Unmute speakers. Enjoy.

So far this works for the tech demo. I’m going to try it on Alpha later.

UPDATE: It works on Alpha. Yay!

Mount USB devices in Virtualbox-based Vagrant

Wed, 24 Jul 2013 17:03:05 -0700

If you want to mount USB in Virtualbox, you have to do this solution. At the time of this post, a typical Google search for this would go to a solution at advocated using the attachusb command in VboxManage in the Vagrantfile provider customization section. This will not work because the VM is off at the time of bootup. It looks like the original author of this gist did not repost his solution back to the thread.

The solution, in other words, is to use USB filters to automatically connect devices to your VM. USB filters can be added while the machine is powered off. They’ll be applied upon boot or, in this case, vagrant up.

My current vagrant setup

Sat, 22 Jun 2013 20:09:42 -0700

I firmly believe that Vagrant is the quickest way from nothing to a running and preconfigured development environment on any machine and especially Macs. For me, the ‘works on my machine’ problem is the biggest reason I run Vagrant. Day-to-day though, Vagrant is probably the easiest to use UI for Virtualbox. If my work actually had money to give me for the VMWare plugin, I believe it would be a better UI for VMWare Fusion as well.

The Basics

This is enough to get started with Vagrant and to reap the rewards.

That’s great and all, but these are the basics. At the very least, you’ll be able to bring up some boxes that don’t require special plugins up.

Frills

You don’t need these but I do! I usually build my Vagrant boxes with Opscode Chef, a configuration management system. For reference, a Chef cookbook is a series of statements about how a machine should be setup.

Most of these frills are plugins. To build the boxes I make, you’ll usually have to install or use these.

Berkshelf is a dependency resolution manager for Chef. I use Berkshelf as a gem along with the corresponding vagrant plugin. With this, when I run berks cookbook <name>, I can make a Virtual Machine that can be created and destroyed quickly from scratch for whatever purpose. I could do vagrant init but berks cookbook has it beat by creating a directory structure that’s pretty much a Chef cookbook. Even if I don’t intend to redistribute said cookbook, the VM made is perfectly fine for tryout purposes.
I like using the Opscode Bento boxes. They are minimal and they have already been uploaded to S3 on Opscode’s dime. In a Vagrantfile, you can set the box URL for a Vagrant basebox to be downloaded. These boxes are great.
You can’t use the Opscode Bento boxes without the Vagrant Omnibus plugin. Those boxes do not include Chef so you must install Chef at runtime.
Just so you don’t get warnings about the Virtualbox additions being out of date, there’s a Vagrant plugin to automatically update the guest additions if needed. This one is really optional and it’s use just surpresses that warning you get if you bring up a vagrant box with old guest additions.

The Future

In the future, I’ll like to be able to test my boxes to make sure they stay working as the things they pull from the internet change. For this, there’s Test Kitchen.

Unfortunately, it’s still really cutting edge. However, there are guard plugins and this Youtube Video and blog post. That video is very much a must see for anybody who appreciates TDD.

Full integration testing on your own laptop is very attractive to me. I keep mine plugged in and I find it disturbing that the rest of the cores on this MacBook Pro just lie cool.

And also, maybe if I get some cash, I might drop some money on the VMWare plugin and VMWare Fusion. If I want to simulate multiple servers at a time and heavier loads, it would make that much faster.

A Windows "QuickStart"

Sun, 26 May 2013 20:33:11 -0700

I paid Swish $25 and $8 shipping for this two months ago. I think I had my share donated to the Khan Academy.

It’s the “Windows Quickstart Kit for Mac Developers”. It includes:

A full Windows 8 Pro license
A Parallels Desktop license
A USB Stick with an ISO on it with a real Microsoft Certificate of Authenticity. Genuine Windows!

I’m not sure why Microsoft saw fit to spend two months packaging a “quickstart”. Why Microsoft did not see fit to provide this quickstart through digital distribution is beyond me. They could have emailed a link to some keys and ISOs and maybe even let a distributor like Digital River do this. Instead, they saw fit to contract this work out to a physical media distribution company. It seems with these charity-ware deals that the best way to donate money is to allocate a large chunk to some physical distributor.

I’m so stoked to try and develop for Windows Phone. I’m going to virtualize Windows 8 and develop a native app for a platform on which I have no way to run natively. I’m also going to jump over some chairs in celebration.

Nuh uh, virtualization is cheating and is really anti-dogfood. If I have to boot up a RAM hungry OS to RUN the app, it’s a failure. If I have to develop the app inside a virtualized OS, that would be even more of a failure since I would have to bear with the performance and non-nativeness penalty. There’s no way a good app can come out with so many barriers like that. Google realized that its Eclipse toolkit was a problem and switched to IntelliJ IDEA to reduce their barriers and gain an awesome IDE in the meantime. I highly doubt Microsoft will provide a free cross-platform and light SDK. The technical barriers are just too high. It’s totally not like their Xbox where game developers are used to putting up with shit like that. Mobile developers are in general more finicky and prefer native tools. I’m not sure if this is still the case now but the Android team saw fit at one point to maintain the Android compilation toolchain for the entire OS on OS X. You can compile a Linux system on OS X!

With that said, this cool cardboard box is all that remains. It would be well designed if it never existed. It’s way too late for that though.

It’s made up of two big pieces. After ripping off the plastic wrapper, there’s a windows logo cutout sleeve thing to keep the box with a flap closed. It’s a pretty retail package for something that is only sold online.

When you open the box, you have a welcome card with some very basic instructions on the back on the left and some coupon card things on the right.

Beneath the welcome card is a USB drive. It has a Microsoft certificate of authenticity on it, so you know its real and has real Microsoft binary bits. There’s also a tracking ID on it too. The Microsoft logo is embedded in plastic on the back and not some cheap printer job. It’s certainly done with style to make the USB drive seem to be actually worth something.

On the USB Drive is just an ISO. There are no other files The drive itself isn’t bootable. It is formatted with NTFS though. At the very least, it’s readable on all platforms. It also enumerates as a device of the name Windows 8 Pro in system profiler. Quite a Matroska doll. I guess having an ISO is easier than mounting a USB drive in a virtual machine in terms of instructional material.

The cards on the right had keys on them. They’re business card sized and feel like them too.

On the back were some keys. Why they felt the need to kill all these trees and grow soy beans to produce the ink for printing on said dead tree material to wrap around these numbers is beyond me.

It took two months for these two alphanumeric strings to get here. The lack of timeliness on delivering these two codes and an ISO does not look well for Microsoft’s Windows efforts at all. It would take 30 minutes to illegally obtain these materials. 30 minutes is far less than two months. Oh, I almost forgot, they don’t even include Parallels in the package. You will have to go download that from Parallel’s site. What a “quickstart” indeed.

Here's a compiled version of Keycastr.

Wed, 22 May 2013 13:02:12 -0700

Update: Keycastr is currently maintained at github.com/keycastr/keycastr. Please use that repository first.

I just compiled a version of keycastr and put it here since I could not find a precompiled version that’s relatively new. The release artifacts are available here:

Keycastr releases

Keycastr is used for presentational demos. It’s free and the source is here in this GitHub repo. You will need a modern version of Xcode to compile it.

The version that is compiled is by creemama. I only blindingly updated the settings with Xcode recommendations. This “works for me” on OS X 10.8.3.

Boot Camp, Windows 7, and USB Installation Gotchas

Mon, 13 May 2013 15:23:28 -0700

A few days ago, I decided to pull the trigger and install Windows on my Retina MacBook Pro at a LAN Party. It’s a perfectly fine machine with a decent graphics card and CPU. I started Boot Camp Assistant and followed through with the instructions to install from the USB drive using the ISO of Windows 7 I had on my Desktop. If you follow Apple’s steps to the letter, your installation of Windows will never proceed.

If you do follow the steps, you will most certainly get an error of “Setup was unable to create a new system partition or locate an existing system partition. " when you select the BootCamp partition that you had just formatted in the Windows installer on your Mac. This can occur on PCs too but basically the issue is that the USB drive is initialized as another valid bootable disk. If you were to Google this, you will find solutions online that involve yanking out the USB drive and reinserting it before pressing “Install Now” on the Windows 7 installation screen. This “Install Now” screen is a big centered button that prompts your to install now and the arrow is set inside a blue circle that’s like a gem.

This is where Apple’s helpfulness can get in your way. By default, Boot Camp Assistant copies the Boot Camp support files to your USB drive. It also adds an Autounattend.xml file to the root of your drive to automatically install Boot Camp support drivers and utilities in Windows. With Autounattend.xml present, you will not have the opportunity to yank out and reinsert the USB drive before continuing. Instead, the setup just proceeds as if that button was pressed. While this is a good file for disc installations, it causes a massive problem for USB installation which is most likely to be the case for all new Macs from now on since they do not include disc drives by default. If you try to ask Apple to ‘fix’ this, they will just blame Microsoft which admittedly isn’t blame free either.

To solve this problem, move or delete Autounattend.xml on the root of the USB drive and proceed with the normal steps afterwards . After Windows is done installing and since Autounattend.xml was nullified, you will have to run the Boot Camp support installer from the BootCamp folder on the USB drive.

It’s a shame Apple’s instructions just do not work. The fix required would probably mean removing Autounattend.xml functionality for USB Drive installations. In the meantime, this worked for me.

Reddit Link: LEARN VERSION CONTROL if you are in UCSB CS!

Tue, 07 May 2013 16:19:28 -0700

Here’s a comment I wrote on /r/UCSantaBarbara a month ago. Learning Git is a very valuable tool for any aspiring or lazy programmer.

UCSB CS FAQ comment thread

At my current place of somewhat employment, the candidates for our internship offering job did not know how to use Git. I hope my post will have changed something.

If you’re new to Git, git-scm.com is still a good starting point.

And suddenly, nearly a decade later, a blog appears! Again.

Sun, 14 Apr 2013 22:05:53 -0700

I’m ashamed to say I’ve neglected this domain and site for 9 years. In that time I’ve gone through high school and university in that time. In each period, I’ve attempted putting up blogs, forums, another blog, and other similar nonsense. I did not follow through with whatever resolutions I may have made to keep the site maintained or write down the latest results from my last screwing around with my projects. Basically, for the past 9 years, I don’t really have much to show for it other than being able to rattle off some helpful anecdote from experiences I have never written down here and there.

Remember kids, the only difference between screwing around and Science is writing it down.

– Adam Savage, Video

I thought I was conducting Science for the past few years; in reality, I’ve only been really screwing around.

Most of my experiments in the past few years have also ended in spectacular failures. I never wrote about my failures even during the times I had a blog available. This is where my fails turned into an epic fail. I recently watched a TED talk on this and learned that this phenomenon is called research bias. I should not have withheld what I had learned from failing. My memory of my failures only fade away until someone mentioned it and I rattle off some anecdote. This obviously won’t scale and does not do much to help me demonstrate any skills when I can’t even get my foot in the door.

I’m going to change that with the launch of my new site. I’ll be honest here, this is very much a career move. However, it is also a personal one for reasons I’ve outlined above. I can’t let those many hours of screwing around go to waste! Prospective employers and/or clients love to hire people who do Science, not screwing around. And I think this site will fix that.

My new site is designed is designed so much as to be a portfolio and as a blog. My portfolio is small as I have little to show and my blog has a meager amount of posts. As a foundation though, the site is solid and is totally ready to showcase more of my skills.

Because of Hope

I started the project to fix up my personal site in September 2012.

At that time I had heard about this static site generator called nanoc. I had taken down previous versions of this site since I could not be bothered to maintain the PHP beast that is called Wordpress. Also, there was little hope of me hacking on Wordpress to change its default style to something that better demonstrates my skills. Stock Wordpress themes just do not exhibit confidence in web development. Also, learning PHP is just something I’me not too interested in doing. The horror stories of PHP may be FUD, but I’ve experienced and witnessed them myself in a past lifetime. nanoc answers both of these problems. I can hack Ruby and static HTML pages on a server do not have moving parts. nanoc is written in Ruby and generates static HTML pages for uploading onto a server. Compared to a beast like Wordpress, it is also very transparent. The nanoc guys can better explain their kind of product (static site generators) here. I did have a problem in that I had absolutely no experience with nanoc.

In September 2012, I answered a request for help from an organization named Because of Hope at the beginning of my last quarter at UCSB. Because of Hope is an organization that helps children and women in Uganda through sustainable methods. Here’s the email that was sent to the UCSB Computer Science jobs list:

My name is Natalie Lemonnier and I graduated from UCSB in 2011. Since graduating, two of my friends and I have started a nonprofit here in Goleta called Because of Hope (BOH) that is working to empower women in Uganda. I am emailing you because our nonprofit is getting ready to launch a new Student Sponsorship campaign, which will require making some changes to our website, and I was wondering if you knew of anyone (student, faculty or otherwise) who might be interested in helping us make these changes. I am not sure if websites are your area of focus, but I just wanted to contact you and see if you could point us in the direction of someone who might be able to help.

Originally, our friend built our site from scratch. However, he is no long working with us and none of us have any web background to make the necessary changes. I believe that some of the fine-tuning will be relatively easy, while other issues may be more time intensive.

However, because we don’t have web background, we are not really sure of the exact time needed to complete the desired changes. If you or anyone else you know would be interested in volunteering to help us, or would like to learn more about our situation, we would greatly appreciate it.

Because we have only been operational for a year, no one in our organization currently gets paid, so unfortunately it’s not in the budget to pay someone to help us with our website. But we are hoping to find someone who would be understanding of our situation and wiling to help us further our mission! I know UCSB is getting ready to start up, so please feel free to pass on this information to students you think would be willing and able to get involved. Please let us know if anyone comes to mind! Thank you for your time, and please feel free to contact me via phone if that would be at all more convenient for you! (818) 620-8847. Happy Fall Quarter :)

(If you would like to learn more about our organization or view our site, please feel free at http://www.becauseofhope.org.)

Best of Wishes,

Natalie Lemonnier

So, I needed some experience doing a nanoc site, something nice to put on my résumé, and something to satisfy and fulfill the “help poor children in Uganda” request in the Vim text editor. Because of Hope needed someone to maintain their site. With my résumé as sparse as it is, I called up Natalie and offered to help.

I met up with them and took a look at the setup. They had a $20/month VPS to serve one HTML file. The rest of the “pages” were actually linked tumblr posts. I told them upfront I wanted to test out some tools on their problem before I ran with it on my own personal site. In the meantime, I would maintain their current handcrafted site and check out any technical details such as the domain name, Google Apps, or the VPS itself.

I had nanoc chosen out, but what about the rest? I checked out the then-current website’s code and it had some custom 980px grid system. I couldn’t be bothered to try and understand it. I did try to understand it enough to add a donations page that they immediately needed for their upcoming fundraiser.

On the bright side, all the mockups that Nikki Day of BOH did were in Photoshop. They were far more ambitious than what the current site’s current implementation. The custom 980px grid system was not up the the task. In retrospect, it would have been thrown out anyway. I shudder at what beast I would have had to slay if the original plans had gone through.

I chose to use Zurb Foundation. Zurb Foundation is a front-end framework that is far more flexible than many other frameworks out there. This was the the kind of Framework that could handle Nikki’s designs. I would also get free functionality such as drop-down menus, pretty buttons, photo carousel, and first-class support for SCSS, an extension of CSS. Zurb Foundation’s use of SCSS was very instrumental in adapting the design of Zurb Foundation for Nikki’s designs. Compared to its competitor Twitter Bootstrap, there is far more control over elements on the page, especially the buttons. Also, I got a mobile site for free! BOH didn’t even ask for it but it came with it!

Ever since from that time, I’ve been working on BOH’s nanoc site. I learned how to better compose layouts, the Rules DSL of nanoc, and the whole nanoc ecosystem. I was also working with it to integrate Zurb Foundation into the project.

Remember that $20/month VPS? It has since been replaced with Amazon S3. I also tossed in Cloudflare in front. BOH can take on a DDOS attack of the most epic proportions. Why would anyone attack a non-profit is beyond me but the overkill just tickles my funny bone. And the cost is free to near pennies! This is something only a static site can do. As a vote of confidence in S3, NPR used S3 for transferring data in its election app. I’m not saying that BOH will become as big as the presedential election, but it can definitely handle that much traffic if need be and certianly more than a $20/month VPS. Honestly, I’m just glad I don’t have to manage a Linux server to achieve this goal. It’s really upload and forget.

Also, in doing nanoc, I now realize how much of a pain in the ass it is to update the original site. There was so much copy-paste! How do people do all these busy work?

I learned a lot from doing the BOH site and I am continuing to work on it in my spare time. It’s source code is versioned in a Git repository so I can experiment on it all I want. It’s also open source in a GitHub repository. If anybody wants to peruse it and reuse parts of it, they are more than welcome to do so. I plan to keep it in sync with anything that is a nanoc best-practice and update it with new pages.

Mindflakes

Okay, so I’ve helped a non-profit get their site stuff in shape and beyond.

What about my personal site?

Ever since I’ve started the BOH site project, I’ve probably made 3 false starts in creating mindflakes.com. I wager I wasn’t confident enough in my nanoc knowledge in the previous starts. I had given up or I did not have enough time to really sit down and really start work on my site. Also, I was working part time at a local company in the area at this time. I came back home so exhausted from looking through piles of Ruby and Python that I really did not want to put my finger to the keyboard.

Maybe the fourth time was the charm? I didn’t start the mindflakes project this time. What I did instead was start an example/bootstrap project for a blog that used Zurb Foundation and nanoc. I put some of the best-practices I knew into there and learned some more best practices from various sources around the web. I’m especially proud that the project is completely tied to Zurb Foundation as its Javascript is sourced from its gem. There were plenty of Bootstrap examples out there but I believe that Zurb Foundation better fits with the nanoc philosophy in letting power come out in the right places when needed.

By distilling the framework from the content of my site, I was able to work away from any pressure in having a nice site to present. Just having a good framework to build upon is much more gratifying.

Not only that, I now have a free reusable framework that I may be able to reuse for client projects that don’t involve my site. I don’t have to extract magic or best practice out of my site. I can just rebase anything off of the framework with the way I like it. I plan to contribute anything I think is relevant while building my own site back to the framework.

This is my first long blog post since ever. I wrote this to test out this blog. As of this post, Mindflakes.com is still very much out-of-the-box even though I just built the box. I will customize it and make it look pretty enough to really showcase my work over the next few sessions. I’m going to have to decide what to do about images and what to do with the flexibility that Zurb Foundation offers me.

As for what to write, I think I have that settled.

Nelson Chen's Blog

Bambu Lab Store Filament Tracker

Where the frustration started

What the tracker does

Building it fast (and keeping it moving)

Things that happened after launch

Backend at a glance

Surviving a power outage with fiber ONT and PoE setup (Frontier FOX222)

Talky Pet Watcher, a cat watching AI

comma three faux touch keyboard

🦾 comma three Faux-Touch keyboard

comma three faux touch keyboard

🦾 comma three Faux-Touch keyboard

Replicate.com openpilot Replay Clipper

Colorized-Project: A Project-Color restoration.

Colorized-Project

GitHub Wiki SEE: Search Engine Enablement: Year One

openpilot Replay Clipper

Datasette-lite sqlite-httpvfs experiment POC with the California Unclaimed Property database

openpilot Replay Cliper

Released Gargantuan Takeout Rocket

Gargantuan Takeout Rocket

Cell Shield

Why I built this?

Popularity

Maintenance

GitHub Wiki SEE: Search Engine Enablement

What’s the technology behind it?

Supercharging it

The results

Things I Learned

GitHub Wiki SEE

The short lived life of Breakout Bots for Zoom Meetings

Technical and Learnings

Reverse RDP into Windows on GitHub Actions

Stop Sims 2 Purple Soup or Pink Flashing Crashes on Windows 10 and Modern Hardware with D9VK

Fast and lightweight headless Qt Installer from Qt Mirrors: aqtinstall

Non-encumbered Windows Bonjour SDK DLLs and Libs

Barrier CI System WIP

An Android Accessibility Service for Eventbrite Organizers: PassShout

Implementation

NC QEMU

Links

Changes

About

Why QEMU+WHPX for x86_64

Packer WHPX workaround

References

Gooey and PyInstaller for easy to write, distribute, and use scripts!

Switched to Hugo

Running Windows XP using Libvirt inside Ubuntu inside VMWare Fusion

BeamNG DRIVE on Mac OS X with CrossOver

Mount USB devices in Virtualbox-based Vagrant

My current vagrant setup

The Basics

Frills

The Future

A Windows "QuickStart"

Here's a compiled version of Keycastr.

Boot Camp, Windows 7, and USB Installation Gotchas

Reddit Link: LEARN VERSION CONTROL if you are in UCSB CS!

And suddenly, nearly a decade later, a blog appears! Again.

Because of Hope

Mindflakes

Why QEMU+WHPX for `x86_64`