Arsenal 2 April 2021 Update
Hey everyone, Ryan here.
One of the things that makes Kickstarter unique is the relationship backers have with the projects they support. We are always committed to transparency with our backers, for both the good (updates on engineering milestones or software enhancements) and bad (delays, or bugs we’re working on).
Today I have some bad news: we’re pushing our shipping target back from June to August.
First, I want to say that the team made absolutely heroic efforts to keep to the original timeline. None of us are thrilled to delay a product launch that everyone is so excited about.
When we launched our campaign, we thought we had a good handle on the challenges COVID-19 would present us. To some extent we did: we’ve been able to work around being a fully remote team and being limited with less travel to our manufacturer than usual. However, we failed to predict a major supply chain disruption lasting into late 2021. We’re not here to make excuses, but if you’re curious as to the details, keep reading.
Parts Availability
Our big challenge is that the lead time on some of our critical components has jumped up by a lot. The worldwide chip shortage, has caused lead times on some components to jump up by months. Normally we could pay a little extra to get the outstanding components sooner (called a ‘Spot Buy’), but currently even that's difficult to find given demand is so high. The companies that would normally sell parts at a shorter lead time (and a much higher price) are also out of stock.
To get a bit further into the details, we did complete the Arsenal 2 Standard Pilot Run because we were able to secure a complete Bill of Materials (BOM) for a small number of units. However, the balance of our purchase order has various parts outstanding before we can complete the production runs.
So, why don’t we change the parts? The CPU has proven the hardest part to source, and if we were to look at alternatives we’d have to go back to the beginning on testing, board design, new coding, and another regulatory approval process - which also adds up to a big delay. We’ve also run into issues with the RAM, Accelerometer, USB Controller, and LED Driver, though we’re more optimistic those will resolve faster and/or will take less rework to swap parts.
We’re doing everything we can to speed up the process and we’re in line to receive parts from a variety of sources. Based on what we’re being told we believe we’ll find the necessary parts within the next 6 weeks. That’d put us on track to begin mass production in June, finish in July or early August, and ship in August.
Remaining Risks
There’s one other issue we’re working on that could impact the schedule, and that’s shipping timelines. We ship to our backers from fulfilment centers in major hubs all over the world, but before those localized shipments go out we have to first get Arsenal 2 from the manufacturer to these warehouses via freight shipments. And freight logistics are backed up due to COVID-19.
Right now there’s a perfect storm of increased demand (companies ramping production for a post-COVID economy) and decreased supply (COVID slows down shipping processes) that is delaying shipments 1-2 months. This may not be an issue when we go to ship, but we wanted to make you aware of it in case it is.
The team and I will keep working our hardest to get Arsenal 2 to you as quickly as possible. We’ll also use the extra time to improve the software and testing, which never hurts when rolling out a new product.
Even though we’ve had the delay on the parts side, the hardware and software team has been making some great progress on things and we do have some good news to share.
Regulatory
Since the last update, we had to make a last minute antenna design change on Arsenal 2 Pro and then quickly ship off units for regulatory testing. I’m happy to say Arsenal 2 Pro has passed all regulatory testing, and we’re in the process of finalizing the reports and paperwork to submit to the NTCBs. (And if you missed it, Arsenal 2 Standard passed a while ago.) Thanks to Phillip for all of the hard work on this.
Banana Stand with Arsenal 2 Standards
Now that we have pilot run Arsenal 2 Standard units, we’ve been able to fully populate our automated test rig (the ‘banana stand’) with Arsenal 2s. This lets us run our test suite across every camera we support.
Software
The software team and I have merged in a ton of code in the last few weeks. Some of our longer running refactors/projects are finally starting to land.
Asset Management
After you take a photo (or stack, timelapse photo, etc..) on Arsenal, there’s a fairly involved pipeline that pulls down the photo, generates preview images, does stacks, and now writes files to the microSD card or your phone depending on your settings. This whole process has a lot of constraints, things need to happen in parallel, but not run the system out of RAM at any point.
On Arsenal 1, each time you triggered something (a stack for example), it would have to wait until all photos from that stack were written to the camera's SD card before you could trigger another shot. The old asset manager needed to wait for the paths of those files to come in so it could associate those paths with the assets internally. Then once those assets were created, the asset manager would start pulling down the photos, stacking, etc.. Ben has been working on a large refactor that lets you run back to back stacks, single shots, etc.. without waiting for the photos to save to the card. (Thanks to Ben for all of the hard work, there’s a lot more involved than you might imagine)
The end result as a user is that Arsenal 2 feels a lot snappier and you’re not waiting for the photos to save before you take the next shot(s).
RAW parsers
The biggest bottleneck in our current RAW stacking pipeline is the actual decoding of the RAW files. A single 50 megapixel RAW file takes up 100 megabytes in RAM, plus some extra overhead. We were previously using an open source RAW decoding library to parse the RAW files. Since the library could only parse whole files at a time, and due to RAM limitations, we had to decode each RAW file, then write it back to intermediate files on disk before doing the final stack in tiles. Now that the new Halide stackers are well optimized, the next bottleneck was the decoder.
After learning way more than I ever wanted to know about RAW image formats, I managed to knock out decoders for all of our cameras’ formats in about a month (except CR3, hoping to have that one done soon). I built them from the ground up to allow the decoding to happen in tiles that are then passed to the stacker. This saves a lot of back and forth to RAM and also keeps the RAM usage really low. Our decoder library is also about 5x faster than the open source decoder library we were using (it varies a bit across formats though).
Night Mode
Andrew has been getting in the final UI for our Night Assist feature (and doing a bunch of bug fixes). I’ve also been making improvements to the night focus feature (using an improved focus training set we made.)
Tutorials
We’ve gotten a ton of requests for a tutorial series for Arsenal 2. Lucian, who does video for us, has been out in the field shooting a full overview of Arsenal 2 features. I’ve seen some early cuts and I think it’s going to be a great way to get up to speed with everything Arsenal 2 has to offer.
Hardware
Shortly after our last update, we got some Arsenal 2 Pro production samples in from our manufacturer. The sample boards are as close to the final units as it gets before the pilot run. When these units came in however, the antenna performance was not the same as we had seen in our prototype boards.
We always prototype and test on the workbench to evaluate performance and feasibility. The antenna design for Arsenal 2 Pro was no different. Our first release candidate was a custom dual band monopole antenna that was manufactured by stamping and bending a thin piece of sheet metal. It was hand soldered to a single pad on the printed circuit board and supported by a plastic carrier. It performed well on the bench and in field testing. We were ready to go on this design until we received our first set of production samples from our factory in China.
What we noticed was some of them performed really well, some of them performed marginally, and some didn’t radiate at all. After some thorough debugging, we found the cause of the issue was manufacturing variance. Basically, any shift from the optimal position caused the antenna to detune, degrading the performance until it didn’t work at all. The root cause of the variance boiled down to two things; assembly complexity and a design flaw. It was simply too hard to solder these consistently by hand, and the antenna was only soldered to one pad on the printed circuit board, making it very fragile and easy to shift during the next assembly steps. Sadly, no amount of jigging or automated assembly could solve these issues… so we went back to the workbench.
After many long days, and a lot of back and forth with our manufacturer, our second release candidate solves both of the above issues. It is also manufactured by stamping and bending a thin piece of sheet metal, but this one is shaped as a planar inverted-f antenna (PIFA). I’ll spare you the details on the differences between PIFAs and monopole antennas, but needless to say they are shaped and sized much differently.
One advantage of PIFA’s shape is it allows us to solder the antenna in multiple locations, keeping it from shifting during next assembly steps (issue 2 solved). This PIFA is relatively tall, so we had to think outside the box when it came to mounting it to the printed circuit board without changing the existing enclosure design, and we came up with an awesome solution. Basically, the PIFA is soldered to a small printed circuit board, with castellated pads, which we call the “antenna carrier”. The PIFA is placed on the antenna carrier with a pick and place machine and then reflowed, ensuring consistent positioning (issue 1 solved).
The antenna carrier is then soldered to the main printed circuit board. But here’s the kicker, the antenna carrier is soldered to the main printed circuit board upside down! How, you might ask? Well, the main printed circuit board contains a large, beveled cutout, sized for the PIFA to sit in and special soldering pads to keep the antenna carrier in place. Placing the PIFA in a cutout instead of directly on top of the main printed circuit board ensures there are no interference issues between the PIFA and the enclosure. Check out the picture below:
Phillip put a ton of time and effort into this design. The performance is great and it's easy to consistently manufacturer.
Thanks again for your incredible support throughout the launch process. The feedback on these updates has been really great. More to come soon!
- Ryan
P.S. Here’s a photo from Lucians tutorial series he’s working on, and a few from a past trip.