The venerable Voyager 1 spacecraft is experiencing another glitch. Instead of sending science and engg. data, it is sending a 0101 bit pattern.
The problem has been narrowed down to the flight data system (FDS), which is not communicating properly with the telecom unit (TMU). A reboot did not help.
Stay tuned as NASA engrs work out a fix for this 1970's era computer, which has performed magnificently during its long 46-year journey to the planets and to outer space.
https://blogs.nasa.gov/sunspot/
1/n
The two Voyager spacecraft, launched on Sept 5, 1977 and Aug 20, 1977, have been traveling in space for over 46 years.
Voyager 1 is farther away from earth at 24.3 bil km (22.5 light hours), while V2 is 20.3 bil km away, located below the ecliptic. Both spacecraft are in interstellar space.
Here are the locations and some vital stats on the two Voyager spacecraft.
You can follow the real-time status of Voyager at https://voyager.jpl.nasa.gov/mission/status/
Graphic source: https://www.nasa.gov/solar-system/nasas-new-horizons-reaches-a-rare-space-milestone/
#Voyager
2/n
Voyager's computer systems were custom-built using 1960s technology, with clock speeds measured in KHz and RAM in kbytes, running hand-crafted software, crammed into 4K of 18-bit wide plated-wire memory (similar to but better than core mem).
And yes, it uses digital 8-track tape for storage.
The custom-designed hardware, (upgraded) software and instruments are mostly still functioning after 46 years in space!
https://history.nasa.gov/computers/Ch6-2.html
https://hackaday.com/2018/11/29/interstellar-8-track-the-low-tech-data-recorders-of-voyager/
@NSFVoyager2
#Voyager
3/n
This schematic of the Voyager telecom system shows that the FDS sends data to the comm system over 2 serial interfaces - a low rate 10 b/s interface routed to the S-band transmitter and a variable rate 10 - 115.2 kb/s interface whose bits are sent via X or S band.
Also, from the 2 diagrams (this post and post #1), the outer coding (Reed-Solomon) is done in software!
What do you think might cause the data to be stuck not at 0 or 1 but at 0101?
https://descanso.jpl.nasa.gov/DPSummary/Descanso4--Voyager_new.pdf
@destevez
#Voyager
4/n
For those interested in failures and recovery in far away spacecraft, check out this thread in August, when Voyager 2 lost contact with earth due to a mispointed antenna (caused by operator error ).
https://fosstodon.org/@AkaSci/110831401826701180
#Voyager
5/n
Richard Stephenson of DSN Canberra explains on twitter how NASA verified that the uplink is working.
They sent a command to Voyager 1 to switch between non-coherent mode and coherent mode transmission.
In coherent mode, the Transmission clock is derived from the Rx signal instead of from the AUX oscillator. This changes the Tx RF frequency a bit which was detected at the DSN.
https://science.nasa.gov/learn/basics-of-space-flight/chapter10-1/
https://descanso.jpl.nasa.gov/DPSummary/Descanso4--Voyager_new.pdf
#Voyager
6/n
In the blog post at https://blogs.nasa.gov/sunspot/, Voyager engineers point out the difficulty in diagnosing problems and crafting solutions for a spacecraft with a signal round-trip-time of almost 2 days and hardware/software developed over 46 years ago using technology long since obsolete.
"Finding solutions to challenges the probes encounter often entails consulting original, decades-old documents written by engineers who didn’t anticipate the issues that are arising today."
#Voyager
7/n
NASA DSN in Goldstone, CA is currently receiving the downlink from Voyager 1 at a reduced rate of 40 bps. No uplink at this moment.
Apparently, Voyager 1 switched data rate (160 -> 40 bps) & did a full memory read-out of her Attitude and Articulation Control Subsystem, Flight Data Subsystem, and Command Computer Subsystems A&B.
Transmission time = 6 hours
Download size = ~108 kBytes
Here's hoping that the received data is not 0101...
A similar but not identical problem afflicted Voyager 2 in 2010. Received science data (but not engg data?) was garbled.
The problem was traced to a flipped bit in the program stored in the FDS. A command was sent to flip the bit.
The issue was diagnosed by downloading a full memory image, which implies that engg data download was working.
This is probably what was done today with Voyager 1 today. Hopefully, it is a similar problem.
https://voyager.jpl.nasa.gov/news/details.php?article_id=16
@destevez
#Voyager
9/n
NASA did not provide a date but it looks like this issue was discovered and acted upon on Dec 7 or 8.
The graphic below shows the schedule for Voyager 1 comms via DSN, generated on Dec 7. Normally, the downlink rate is 160 bps. On Dec 8, it was switched to 40 bps. And again on Dec 10. Some special commands for the FDS were also sent.
Since then, the D/L rate has been switched between 160 bps and 40 bps a few times with additional FDS commands uploaded.
https://voyager.jpl.nasa.gov/pdf/sfos2023pdf/23_12_07-23_12_25.sfos.pdf
#Voyager
10/n
Two-way comms happening now between Voyager 1 and NASA DSN Canberra.
Of course, the results of the uplink commands will arrive 45 hours from now. The data arriving now left Voyager 1 22.5 hours ago.
Downlink rate is the lower 40 bps rate.
The DSN schedule for Voyager 1 shown below was modified and published yesterday.
Here's hoping that Voyager engineers are getting closer to a solution
https://eyes.nasa.gov/dsn/dsn.html
https://voyager.jpl.nasa.gov/pdf/sfos2023pdf/23_12_14-24_01_01.sfos.pdf
#Voyager
11/n
NASA JPL provided a minor update today about the status of the Voyager 1 spacecraft, indicating that the comm. problem that started more than 2 months ago has not been resolved yet. No other details.
Please check out the rest of this thread for more info on the problem where instead of sending science and engg. data, Voyager 1 has been stuck sending a 0101 bit pattern.
@NSFVoyager2
#Voyager
12/n
No new info on the status of the Voyager 1 spacecraft, which since Sep 2023 has been sending a 1010 bit pattern instead of real data.
Several popular science outfits have been covering it lately. A bit flip in the FDS is suspected, but it is difficult to identify since the memory cannot be read back.
Several commands were sent yesterday to Voyager 1; responses will arrive 45 hours later tomorrow.
Wonder why they cannot overwrite all prog and data memory.
https://arstechnica.com/space/2024/02/humanitys-most-distant-space-probe-jeopardized-by-computer-glitch/
#Space
13/n
Good news from the Voyager 1 spacecraft that has been stuck sending a 0101 pattern since Nov 2023.
The team has long suspected the root cause to be a corrupted area of memory in the FDS computer. On Mar 1, they sent some commands to make the FDS skip around sections of memory. The data stream rcvd 45 hours later looked different and was decoded to contain a read-out of the entire FDS memory!
Hopefully, they can now identify and fix the offending memory words.
https://blogs.nasa.gov/sunspot/
14/n
Voyager is not out of the woods yet, but the lesson for all of us is to never ever give up.
Here is the schedule for comms with Voyager 1 via NASA DSN this weekend. Some new commands will be sent on Friday, with responses expected 45 hours later on Sunday.
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_03_14-24_04_01.sfos.pdf
15/n
Some tech. info on the Voyager FDS computer –
- There was a backup FDS unit but it failed in 1981.
- Custom CMOS CPU - 36 instructions. 80 KIPS, 115 kbps data rate.
- 128 registers, kept in memory.
- CMOS memory, a first in space, 8KB.
- No separate memory for program storage vs execution. The CMOS memory is non-volatile kept powered on by the RTG.
- DMA access to memory by hardware. Instead of “cycle-stealing”, the instructions indicated cycles where DMA can occur.
https://ntrs.nasa.gov/api/citations/19880069935/downloads/19880069935_Optimized.pdf
16/n
Status update on the Voyager 1 spacecraft which has been sending a 0101 pattern since Nov 2023.
The problem seems to be a failed memory part in the FDS computer; engineers are planning to move ~200 words of software from one region to another, according to Joseph Westlake, director of NASA’s heliophysics division, who was speaking at a March 20 meeting of the National Academies’ Committee on Solar and Space Physics.
Westlake sounded very optimistic.
https://www.nationalacademies.org/documents/embed/link/LF2255DA3DD1C41C0A42D3BEF0989ACAECE3053A6A9B/file/D727AF88E8C806D7A1F75C8401AF9CF23BCCC2EC9F3A?noSaveAs=1
17/n
It looks like the Voyager team is preparing for a new "memory upload" to the FDS computer on Friday, as evident from the DSN schedule and instructions shown below for Voyager 1.
I am guessing that this is to rearrange the software so that it no longer uses the locations in the faulty memory chip in the FDS. If true, then hopefully we will hear Voyager 1's true voice on Sunday, 45 hours later. OTOH, this may be just one of many steps on the road to recovery.
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_03_28-24_04_15.sfos.pdf
18/n
Looks like the "memory upload" to the Flight Data Subsystem (FDS) on Voyager 1 is taking place at this time from the NASA DSN site in Canberra.
Go Voyager!
https://eyes.nasa.gov/dsn/dsn.html
https://en.wikipedia.org/wiki/Canberra_Deep_Space_Communication_Complex
19/n
It's been 6 hours since the "memory upload" data was transmitted to Voyager 1 from the NASA DSN site in Canberra.
During that time, the signal has traveled about a quarter of the way to Voyager 1, about the average distance to Pluto. The response will arrive at earth on Sunday around 1500 UTC (RTT = 45 hours).
Let's imagine a spacecraft sent to the nearest star Proxima Centauri, 4.2 light-years away. How would we diagnose problems and upload new software to it?
All eyes and ears on Voyager 1 as data is being downloaded from it in response to the "memory upload" commands and data sent 45 hours ago.
The DSN Goldstone 70m antenna is receiving data now at 40 bits/s.
Hopefully, the problem with its transmission being stuck at 0101 has been fixed. We will find out this week ...
NASA Voyager twitter site confirming that Friday's memory upload was intended as a fix for the Voyager 1 transmission problem caused by the failed memory chip in the FDS computer.
"Sister @NASAVoyager's reply to Friday's upload should b arriving now @ Goldstone's 70m antenna DSS-14, hopefully confirming that the Flight Data Subsystem memory update was successful. If so, telemetry should now give clearly interpretable signals with science & engineering data!"
22/n
Great news on Voyager 1.
Richard Stephenson of DSN Canberra reports that engineering data was being received from Voyager 1 last night at 40 bps.
No science data yet, perhaps because they did not switch to the higher 160 bps rate, but this is a major step towards recovery and validates the diagnosis (failed memory chip in the FDS computer) and fix (rearrange software to bypass the failed memory area).
Now waiting for a status update from NASA.
23/n
Minor update from the NASA Voyager 1 team -
- Confirmation that the problem is due to a faulty memory chip, which affects about 3% of the FDS memory. We have known about this diagnosis since March 27 but if was not officially announced until today.
- No info on the recent report that engg data was received last Sunday.
- "It may take weeks or months" to fix the problem so that Voyager 1 can operate normally without the unusable memory hardware.
The DSN schedule for Voyager 1 next week shows commands to be sent on Tuesday to relocate some FDS code around "VIM5". Presumably, VIM5 is a memory module.
Additional uploads will take place on Thursday.
So, there is still work that lies ahead to rearrange the code around the failed memory chip. And we suppose, it has to be done incrementally and meticulously so as to not accidentally brick the FDS computer.
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_04_04-24_04_22.sfos.pdf
25/n
The Voyager 1 team sent some more commands today to the spacecraft to relocate code around the failed memory area in the FDS computer. The command will reach Voyager 1 tomorrow, 22.5 hours later; we will know the results on Saturday.
The tweet from the Voyager team indicates that the software being relocated is the "program for proper encoding and compression of engineering data (EL-40)."
Would an analogy of replacing spark plugs on a running car be accurate?
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_04_18-24_05_06.sfos.pdf
26/n
The NASA DSN site in Canberra is currently receiving data from the Voyager 1 spacecraft. The data will ascertain whether the commands sent on Thu to relocate code around the failed memory area in the FDS computer, worked as expected or not.
Some additional commands are scheduled to be sent today contingent on the results, although the uplink is active now.
Here's hoping that we get good engineering and science data from Voyager 1 today
https://eyes.nasa.gov/dsn/dsn.html
https://voyager.jpl.nasa.gov/mission/status/
26/n
Hallelujah! And Congrats all around.
JPL just announced that engineering data was successfully received from the #Voyager 1 spacecraft on Saturday April 20, the first time since Nov 2023.
The commands sent on Thu to relocate some code around the failed memory chip in the FDS worked as expected.
Science data is not being received yet; it will require relocation of some more code in the sparse free memory areas in the FDS.
(Engg data = spacecraft health data)
https://www.jpl.nasa.gov/news/nasas-voyager-1-resumes-sending-engineering-updates-to-earth
27/n
DSN in Canberra should be receiving a full memory read-out of the Voyager 1 FDS computer as we speak. This will help with the planning for further code relocations to enable transmission of science data soon.
Six months after it suffered a serious brain injury and after months of mind-boggling ultra-long-distance surgery, the Voyager 1 spacecraft walked and talked at full data rate today!
After transmitting a full memory readout on Friday at 40 bps, Voyager 1 switched to the science-mode 160 bps rate, which presumably the DSN site at Goldstone was able to receive and decode today.
Congrats and kudos to all who made it happen.
https://eyes.nasa.gov/dsn/dsn.html
29/n
For those curious about the downlink data rates used by Voyager, it uses 40 bps for engg. data and 160 bps for science data. It requires use of the 70m antenna at the 3 DSN sites. A 1.4 kbps rate is supported with 5 arrayed antennas at Madrid.
Voyager 1 is 24 billion km away. Tx power = 23W. Rx power = -160 dBm (1e-19 watts). A WiFi signal is > 10 billion times stronger!
Earlier data rates (kbps) for V2:
From Jupiter: 115.2
Saturn: 44.8
Uranus: 29.9
Neptune: 21.6
https://descanso.jpl.nasa.gov/DPSummary/Descanso4--Voyager_new.pdf
30/n
Voyager 1 is now communicating via the 70m antenna at the NASA DSN site in Madrid, still transmitting at the 160 bps "science mode" rate.
Steady as she goes!
@AkaSci wild to think of an entire code load, just flying through space. Transmission complete, it's just flying through space until a little antenna snags a bit of the energy.
@AkaSci If using a laser, I think it'll widen out so far and be so faint as to not be detectable without adding too much equipment for a small craft. It might make sense to create repeaters. It would still take 4.2 years to get there, but the craft would get a better signal regardless of the source, barring some breakthrough. With some AI in the repeater, it could initiate repairs. That would make common repairs possible without awaiting a signal from Earth. So, yes. With a little help.
@AkaSci with so many things going badly for humanity right now, it feels good to know we can still keep our most distant travelers working a little bit longer.
@AkaSci hats off to the team. This has been an incredible story to follow so far.
@AkaSci looks like it worked! V1 is sending engineering data. Word from Canberra DSN below
CI/CD old style
@AkaSci I’ve been waiting for this all day! I’m eagerly awaiting your next update. Meanwhile I’ll stare at the squiggly line on the DSN site and hope.
@AkaSci Extraordinary!
OK nerds: who wants to create a computer game in which you are the engineer trying to debug Voyager with a half-day lightspeed delay on each move? Turn-based (one move every 12 hours); you have a spec for a very (very) simple microprocessor and some fault data to start with, and your job is to figure out what's gone wrong and patch it. Sits squarely in the genre of sit-and-think puzzle games and https://tomorrowcorporation.com/humanresourcemachine, and would be a fun alternative to Advent of Code if it's ready for December.
Sounds like an awesome riff on TIS-100, get the Zachtronics guys on it maybe.
@gvwilson Set it up like a "free-to-play" game that's going to charge you to play more than one turn per day, but then don't actually let people buy additional turns. You're just stuck with what you get.
@gvwilson only if it supports PVP and I can combine thousands of damage types and environmental factors over thousands of logic gates to create a riddle...
@gvwilson
Voyager 1 is about 23 light hours away, so it would actually be a 46 hour turn around time….
Don’t forget that false moves can brick the device you’re debugging.
(One move every twelve hours will sound familiar to those who started their programming experience by submitting batch jobs.)
@AkaSci for me who is pondering a new smartphone every 2-4 years – considering myself as testing the life expectancy of modern, well tested technology – it is beyond my imagination what technowizardry is happening there.
Not only that antiquity working so long at all, but now being necromanced back to life?
@AkaSci i guess it will require a full reprogramming to avoid allocating memory in the faulty chip. Do they still have the ability to assemble code for the FDS?
@f4grx
It depends on whether the faulty section contains instructions or data. It will be easier if it is data.
Many moons ago, I patched a binary stripped cc executable (before there was gcc) to relocate its symbol table (which was overflowing) to a new area beyond "_end" with a larger size. Thankfully, references to the table and its size were in just a few places in the code.
@AkaSci it depends if they have the assembly listing to regenerate the software load or if they need to do binary patching!
@f4grx
Pretty sure the Voyager team has the assembler and do the programming in assembly language. You and I could write such an assembler in short order.
@AkaSci Sounds a bit more like reworking the timing of a four cylinder car after a sparkplug failed so that it will run smoothly on three cylinders by skipping that one cylinder and its failed spark-plug all together.
@AkaSci I'd say the better analogy is swapping a blown gasket without opening the hood. Its really peak spaceflight operations. Much more if it works.
@AkaSci analogy is whatever this is. Except everyone would also be blindfolded
@AkaSci what does FDS stand for and how does it compare to the AGC? My brain reads “Famicom Disk System” and I know that’s not right lol.
@cameron_talley
Take a look at some of the other posts in this thread, E.g., #1 and #18.
Reprogramming a 46 year old computer from over 15 BILLION miles away. Pretty awesome!
"You sir are a steely-eyed missile man/woman."
@AkaSci Such an amazing, wonderful news! :-)
@AkaSci congratulations to that awesome team of developers! What an amazing achievement
@AkaSci Thank You for sharing. Please continue!
Awesome work at NASA!
I still bet it's a faster download then I'm getting at work today.
Your chart is ready, and can be found here:
https://www.solipsys.co.uk/Chartodon/112331905604076812.svg
Things may have changed since I started compiling that, and some things may have been inaccessible.
In particular, the very nature of the fediverse means some toots may never have made it to my instance, in which case I can't see them, and can't include them.
The chart will eventually be deleted, so if you'd like to keep it, make sure you download a copy.
@AkaSci > After a full memory readout today at 40 bps, Voyager 1 switched to the science-mode 160 bps rate
my wifi be like
@AkaSci don’t anthropomorphize the robot, it hates that.
@AkaSci that's awesome!
@AkaSci It continues to be the little spacecraft that could.
@hans
It's not so much the technology as the distance. Voyager 1 is 24 billion km away. It transmits at 23W. It takes a 70-meter antenna to receive and decode the signal from that distance. Receive power is -160 dBm (1e-19 watts). A WiFi signal is more than 1e10 times stronger!
As Voyager 1 moves further away, the transmission rate will need to be further reduced.
@AkaSci this is one of my favorite ways to see my tax dollars (pennies thereof really) at work
@AkaSci When it was launched, I don't think we had 300 bps modems available yet. It's amazing they've managed to do all of that on a machine that's approaching half a century old!
@AkaSci I'm more curious as how they are able to get the signal from the noise. At that rx it seems like it would just be lost to background.
The video is about Voyager2 but that should be close enough
EEVblog 1547 (Part 1) - Contacting the Voyager 2 Space Probe
"A detailed explanation by Richard Stephenson from the NASA Canberra Deep Space Communications Complex in Canberra on how the 70m DSS-43 dish at the CDSCC is used to still contact the Voyager 2 probe."
@AkaSci hailing frequencies open!