Note: The Sony A7sII now got listed at Amazon US too (Click here)!!!
Yesterday Sony announced that it will add 14 Bit RAW recording on the A7sII, A7rII and yet not defined “additional cameras”. There is one uncertainty highlighted by Diglloyd:
I am puzzled why an uncompressed format would be offered versus a lossless-compressedformat. The core issue is file QUALITY, not file FORMAT. Lossless-compressed is identical to uncompressed in quality, but saves a lot of space.
- An “uncompressed” A7rII file would creat 74MB per file (75% larger than the current Sony 11-7 Bit compression).
- An “lossless-compressed” would create files that are almost one-third less big than the uncompressed version
Some more open questions:
1) Will Sony add the feature on A-mount cameras , first generation A7 cameras and on the A6000 too?
2) What will be the effect on the buffer memory?
3) Will it (and by how much) slow down operations (playback and so on).
Let’s hope Sony will answer those questions soon…
213 Responses
A lot of people on the net kept on using the word “uncompressed”. And although others stepped in trying to correct this to “looseless” many kept on writing “uncompressed” even though they clearly ment “looseless”. Such irony it would be if Sony now gives us uncompressed but NOT looseless raws – instead of looseless!
Hopefully Sony knows what the issue really is 🙂
The issue is simple, Sony wants to shut up the people who have been making this into a massive issue. They are slowly removing all the points people turn into a massive important one. So what sony will do is deliver all the bits it has without changing anything from input to output. The question if they are compressed in transit (thats what lossless compression is) has no effect on removing this from the table. I would not be shocked if it would not be compressed now anymore since that would recover some of the speed/problems/work by not having to add this new step.
They can’t and won’t shut them up because they will just find something else to complain about.
A large percentage of the people complaining about the lossy compression will now complain about the file size being too big for their 10 year old computer to handle. And you’re absolute right, they will find something else to cry about.
No doubt, Sony will have to push harder as the ‘old’ brands on issues its unfair but many people will only accept the perfect camera even if their older ones had their own problems. My guess is they changed this now because the complains started hitting the review sides more and more since all the older complains are gone.
Dpreivew’s article on the lossy compression and the compromises involved really hit home. It was fair and factual unlike some of the more hysterical rantings from much of the blogger and troll community.
At worst the file will be 75% larger than today.
There’s no way around (lossless) compression! With around 75 MByte per uncompressed raw picture and 5 fps we got to transfer 375 MByte/s – the fastest SD cards I know of (that can be used in Sony cams) can write with 95 MByte/s.
The buffer is filled up so quickly that the cam is only usable for landscape pictures or macros but for nothing moving a little bit faster the a snake…
And whether lossless compressed or uncompressed the data is the same – remember what a zip file is?
You just confirmed what i stated on what compression is. True if they just make it uncompressed it means the fps will go down nothing is free in the world. So best would be compressed but lossles but again nothing is free sony went for their older method for a reason its all about balance.
It’s going to be awesome for landscape shooting actually!
Very exciting
It is also good for portraits and anything that does not require continuous high speed. Even bracketing would work – 3 or 5 pictures, then some seconds to write them out. I would likely go for it. My need for multi second high shooting frequency is quite at the zero mark.
Exactly I kept correcting people but it was like swimming against a massive stream… What can I say, YOU GUYS ASKED FOR IT!!! Don’t complain now…
Is whatever you want it to be…!
Just run gzip on your RAWs after you are finished processing them.
That’s a silly idea…storage space is really cheap today on hard disks but storage speed on cameras/cards is still a little bit limited…
So it makes perfectly sense to (lossless!) compress data before storage because of speed!
File size on your PC is only part of the problem. The bigger issue is that transferring data from the sensor to the buffer to the SD card is a slow process. Larger file sizes limit the burst rate and/or burst duration.
This is weird ! Everybody kept asking for uncompressed RAW files , there you go ..
Now you ask for lossless comporessed !
Thank god I’m not a Sony engineer
Sony factor in that most people are idiots, even some that think they are experts. They ignore them, and do the RIGHT thing.
Do you consider the forced lossy compression in Sony’s cameras the RIGHT thing?
I do.
For most pixels in most images the old 11+7 worked perfectly fine and resulted in significant improvement in file size and write speed. The compression artifacts were either non-existent or very small. The artifacts only became noticeable in very specialized cases. So if the Sony engineers were only going to support one option, the 11+7 was the right option.
That said, it will be nice to have that second option for those specialized cases (like astrophotograph) where burst speed is not important but being able to tease out fine details in post-production is.
Your comment at least proves that the first sentence is true.
Not everyone, many of us have NEVER asked for uncompressed and have ALWAYS asked for lossless compressed. Mostly, only those that don’t even understand what they are asking for have asked for uncompressed.
Lossless compression would be cool also, but unlikely necessary.
That said, I’m perfectly happy with uncompressed RAW myself. – can’t wait
I guess they just provide uncompressed raw,
because for compressed files, not matter lossy or lossless, it needs more processing power. uncompressed raw however, actually reduced the workload of processor, the only part to slow it down is it will take more buffer space while saving. so I guess this is how sony be able to add the 14bit uncompressed option without hurting the burst speed. less processing but more saving time make it neutralized with each other
This is right – IF (!) the processor has other things to do. Given the extreme low write speed of the storage side… it may well be that the processor can handle a simple RLE or multi channel RLE encoding. In Programming things sometimes are a little more complex than you want them to be – in this case the write speed may be better when compressing due to the card interface being slow.
What is the write speed on the storage side?
People have measured it at around 45mb/s
Sounds about average.
Larger files will take longer to clear the buffer and write to SD card. They might also add some delay in the picture preview
that would be a really cool trolling of the customers :D. an uncompressed lossy raw, lmao.
Digilloyd is raising correct point: it’s not about format but quality. You can have high quality losslessly compressed.
Lossless compression would be fine as all the data captured would be preserved in the saved raw file. Currently there is a discussion/argument going on at dpreview where some want lossless, others totally uncompressed and some the current compressed lossy raw. To be safe Sony should include all three just to stop the madness if that’s even possible.
If all three become available, the complaint would be that there are too many choices and it’s confusing.
Haha. Sure.
No doubt. 🙂
Not really. Nikon has all 3 choices, and the default is lossless compressed, which is what pretty much everyone uses.
quality
“Then raw”? Never heard of that.
They were always called compressed RAW. Some will remember that earlier Sony (and Minolta) had both compressed (cRAW) and full RAW files (ignoring the brief flirting with noise compressed RAW) and nobody, but nobody seemed to use RAW. Hence cRAW became the standard and it is only in the last couple of years anybody noticed.
“user selectable Compressed and Uncompressed 14-Bit RAW image capture”
The way I read it we should be getting a choice of 2 different files to choose from (possibly along with the existing 11+7 as it seems to be called, which may be needed to keep up the continuous rated frame rates of these cameras), so looks like they are maybe adding more than just uncompressed. They did not mention the type of compression, but as it will be shipping with the A7SII we only have 3 or 4 weeks to wait to find out for sure.
I wouldn’t count on getting it on first generation 7 series cameras, or any of the others that were released more than 6 months ago. Maybe the RX10Mk2 and RX100Mk4 along with the A7Mk2, but going back and re-writing the software for older cameras would be a big endeavour indeed.
>re-writing
it’s a patch.
add on.
new function.
it’s basically copy pasta.
I would bet that they mean 11+7 when they say Compressed. I highly doubt that they would come up with a third format.
On my old Nikon D700, it has been proven that the “lossless” compressed raws were actually a little bit lossy. So uncompressed could seems more reasonable to avoid other problems…
No, it prooves they LIE. Most of my data is compressed lossless (not pictures) and I never switch a bit due to compression. Tools to compress data without loss are around for more than 50 years. ZIP, RAW file formats. What you just say is they lied to you. Finished.
Whut?
“On my old Nikon D700, it has been proven that the “lossless” compressed raws were actually a little bit lossy.” – no, it prooved the lied, not that lossless compressed generally has losses.
Who lied to whom?
Nikon lied that it’s lossless compressed
I have no idea what happens on the D700, but lossless RAW files have 0 loss. Every single bit of the data remains 100% intact.
You may well be right. Nikon were pioneers of a gamma compression scheme which was lossy at the data level, but not at the picture content level. In essence it dropped most of the noise bits as there’s never more than about 9 bits of real information in any part of the image, even in the bright areas (due to shot noise).
There’s a very interesting bit about Nikon’s “lossy” connection in this excellent treatment of the basic limitations of digital photography.
https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html
If your loss is particular noise in every pixel (as opposed to statistical distribution of such noise), it is not really image quality loss. An ideal RAW format would actually be lossy, throwing out the garbage (with a proper RAW processor restoring the noise if needed with the same statistical properties). RAW files would actually become smaller as ISO rises, so both single-shot landscapers and sports shooters would be happy.
If it is uncompressed (still) lossy raw Sony will offer, I must congratulate them for one of the worst marketing stunts ever made in digital photography. I’m not one of those who has been loudly advocating lossless raw, I really cannot see any problems in my photos, and I would not “require” it to be implemented in my A7R and A7S. But even bringing this up, and not be able to deliver what a lot of people actually has been moaning about would be such an epic fail and quite an embarrassment. An embarrassment of harakiri proportions almost.
I really can’t imagine that they’re stupid enough to put out a news and firmware release to offer the exact same information in a much larger file size. I don’t know where this idea that they might do that was hatched (Ming Thein perhaps?), but I’m not really going to entertain it until I see some evidence of it happening.
I agree.
My guess is Sony is adding a mode that will enable a true 14-bit RAW file in it’s purest form. Which is precisely what camera’s like the A7R II would need so as to compete against Nikon who would have undoubtedly milked this particular aspect of performance in their upcoming products.
They won’t. Somebody is just trying to find an escape clause to justify the wording.
However, generating a file that’s larger than is strictly necessary is not unprecedented. Nikon’s uncompressed RAW uses 16 bits per pixel (probably to save processing time) so there’s an example of a file that is large, albeit only about 14% larger, than it need be to hold the data.
With no compression it can’t be lossy. (“lossy” means “lossy compression”)
“Lossy” means data is lost. You could conceivably create a “lossy encoding” function which doesn’t achieve any reduction in size.
Yes and someone at Sony could smash a sledge hammer on every A7 body before packaging. But why should they do that? The only reason to intentionally use lossy algorithms is compression. Did I miss something?
I was just speaking in general, not about Sony. There are other reasons to use lossy algorithms other than compression, in general. But in all likelihood none that apply to this particular case.
Thats what I meant. Speaking in general, you are right, but it’s kind of useless to discuss possibilities which make absolutely no sense, and which have never been done intentionally before, for exactly that reason. Right? 😉
Wrong. There is lossy and non lossy compression. Two different things. I honestly thought everybody had understood that by now.
https://en.wikibooks.org/wiki/Data_Compression/Lossy_vs._Non-Lossy_Compression
both are compression schemes. people don’t intentionally introduce some loss into data without gaining anything. so the loss is due to compression.
Dammit Freddo!!! You are killin me. Read my post! It’s getting more and more confused here.
You said “If it is UNcompressed (still) lossy raw…”. I just said there is no UNcompressed lossy RAW, because if it’s UNcompressed it can’t be lossy! Of course there is lossy and lossless compression, but we was talking about UNcompressed RAW!!!!!!!!!!
Maybe it’s just a spelling error you made in the original post and you meant compressed instead of UNcompressed. I just just pointed that out because so many non tech savvy people get more and more confused.
OK? 🙂
Ok ok, see what you mean now. Sorry. 🙂
It cannot be uncompressed lossy. Loss was coming from their extremely badly designed compression.
I predict this will make Sony’s terrible card speeds much more apparent but one they won’t be able to fix that easily. I suspect the SD card is interfaced with the CPU over USB 2.0.
What is the card speed on the Sony cameras and how did you measure it?
The SD card write speed has nothing at all to do with the USB interface.
“I suspect the SD card is interfaced with the CPU over USB 2.0”
It is not, your suspicion is wrong.
Sony’s card interface does not use USB2.0. No camera manufacturer uses USB for interfacing to a flash card. It would be insane.
I realized I wasn’t clear on this. The SD card reader/writer may be interfaced to the CPU via USB 2.0. Of course the card reader/writer and the card itself with use some version of the SD protocol (UHS-I for modern cards). It is very common for internal memory card readers to use an internal USB interface—at least with computers. I don’t have as much experience with low-level hardware design for cameras but since they are essentially specialized computers at this point I suspect the same is true. I may be wrong but the supposition is far from insane.
It would be ideal if Sony would support both.
I’m sure they will, but not now. It will come in a firmware update for the A7Riv or A7Rv after an online petition in 2018. 😛
They did say that it would be a user-selectable option to use either compressed or uncompressed RAW.
There will be a major effect on the buffer when shooting continuously, because Sony didn’t even use the 14bit compressed for continuous shooting previously — they were only using 12bit.
So if 14bit uncompressed — or 14 bit lossless compressed — is suddenly used for continous shooting, you will see a major effect on buffer.
Take a look at this chart, comparing the buffer of Nikon cameras in different raw settings:
https://photographylife.com/nikon-d750-buffer-capacity
For example, the D750 — in 12bit compressed — 33 images. In 14bit lossess — 15 images.
So I wouldn’t be surprised to see a 50% hit in the buffer when shooting continuously.
Interesting! I wouldn’t have guessed that larger files might affect the buffer. I’d also like to refer everyone to a chart that proves the sky is blue.
In 14 bit raw lossless mode the following features will be unavailable to save processing power:
-Autofocus
-EVF/LCD
Okay, I’m kind of kidding.
Does it even matter?
Turn off for spray and pray and turn on for high fidelity output.
Seems simple enough to me.
When you are stupid enough to underexpose your 10 fps sports shooting by 5 stops, it might matter.
Just kidding… LOL
“There is one uncertainty highlighted by Diglloyd”
It wasn’t exactly highlighted by Diglloyd, and we hardly need him to spot this for us. Everybody has been asking this question since the announcement was made.
I see absolutely no reason why Sony would provide a completely uncompressed format. The current lossy compressed format already proved that there is enough processing power for the compression. So, what would be the benefit of no compression?
No reason, other than “uncompressed RAW” is what people have been screaming for (not fully thinking through what they’re saying), and that would require the least effort on their part. From Sony’s standpoint, the benefit would be that it would leave the basement pixel peeper trolls with nothing to scream about in terms of image quality. It would be great if they could give us all 3… 1) old style 11+7, 2) Compressed 14bit RAW, and 3) Uncompressed 14bit RAW. But my gut says number 2 might be omitted.
Another reason might be to save some “revolutionary improvements” for the next generation Mk III lineup. 😉
No compression would be easier (and thus cheaper) to implement, would require less testing, and would raise fewer questions about whether it was really truly lossless or not.
Yes, and uncomressed raw formats are less prone to unrecoverable damage.
Possible reasons:
1) Because compression takes processing time. That’s probably not a big deal here since the data path is slower than the processor. However, it could be an issue if the compression scheme is complicated.
2) Because compression makes the file vulnerable to defects. In the present system if one byte gets corrupted, it would only mess up one pixel. In some compressed formats, a single byte error could ruin a block, an entire line, or the rest of the file.
3) Public Relations: Sony has faced a lot of criticism because of their existing (lossy compressed) 14-bit RAW. If they released a new RAW format with lossless compression, the critics are still going to be suspicious that the scheme is really lossless. If they put out full 14-bit uncompressed RAW they can squelch any possible criticism about loss of image quality.
Can’t you? Lossless compression is cpu intensive and requires more development. Sony might just release an uncompressed RAW as there’s not time to produce a compressed version. Or possibly the hit on processing power is too much (much in-camera processing, like JPEG production, require hardware assist features or they’d be far too slow and use more power; think how long it takes Lightroom to render a JPEG).
That’s not to say that it isn’t possible. It’s just two reasons why it might not be released, at least in this cycle.
I hate to tell you, but lossless compression does not have to be CPU intensive. It depends totally on the algorithm used. Any sort of RLE compression is extremely trivial on the CPU. High efficiency generic algorithms are not cheap – on CPU and memory. But there are a number of quite CPU light approaches and you can still work with some quite specific algorithm taking. It may not save the same percentage, but it may be a good compromise.
Been there, done that – though not on RAW sensor data.
You can hate to tell me what you like (disclosure, I’d worked a lifetime in IT and have done a lot of communications physics and data theory, so I’m not unfamiliar with the territory). There certainly are very CPU-light compression algorithms, but they are data specific and (RLE especially) probably worse than useless on RAW data which contains inherently high levels of random values due to shot noise, read noise before you even consider the actual subject matter itself.
The more general purpose compression algorithms are compute intensive, and the general processing power available in a camera is limited by, not least by power consumption and the related heat generation. The potential data rates are high to. A 42MP sensor using 16 bit aligned values (Nikon are clearly doing this as it would be computational disastrous to use 14 bit aligned value), requires 84MB to be processed for every shot. That would be 420MBps, albeit that whether this needs to be dealt with in “real time” depends on exactly where in the processing flow this occurred. At write-out time it would not impact on frame rate until the buffer was full, but the buffer will fill up faster (so fewer frames). If it’s done immediately after capture, then the buffer won’t fill up so fast but it might impact frame rate.
Nikon cameras have been designed with the requirement for lossless compression from the very beginning. They may have sufficient CPU cycles and/or they might have hardware assists for compression. We don’t yet know what the position is with Sony’s cameras in this respect. As they are in smaller bodies, with lower capacity batteries and have more demands in the first place (IBIS, full-time live-view) then the power budget may be more restrictive than on (say) a D810.
Note, I’m only giving possible reasons. It could be that the processing chip in the Sony is fully up to the demands, but it is quite possible it hasn’t been designed with the capacity to do lossless compression without a performance hit (or, for that matter, even lossless RAW without a performance hit).
For those that wonder how high-speed JPEG and Video rendering can possibly be performed, that’s all down to hardware assists without which video would virtually impossible and even rendering a JPEG would be painfully slow (time it on a powerful desktop with a CPU with energy to burn). It’s the huge advances in dedicated video and image processing chips that make this all possible. Such features can be added for supporting lossless compression too, but whether it is necessary or not, only the engineers will know.
EXACTLY!
File integrity.
You have to check file integrity regardless if it is compressed or not. File integrity is in danger at any step of information processing. You cant trust any wire, any storage or any processing unit. So even if we see compression as one more danger for file integrity, I would rule this out as a reason.
We need some Sony engineers. They definitely read this forum. They surely could explain it better than me.
I doubt many Sony camera engineers read this site, if any. Most probably don’t read English well enough to enjoy this site.
You are absolutely right that lossless compression is wholly unrelated to data integrity. If you want to guarantee data integrity, you need something like a CRC (cyclic redundancy check) whether the data is compressed or not. Some compressed formats (like ZIP) have it built in and it’s one of the reasons that even incompressible files are zipped.
By coincidence, I read a whole thread about downloads that had been corrupted due to a optical line card fault which passed undetected by checksums in TCP (which has a weak error checking system) and “bricked” some consoles as there was no data integrity check, and that was on uncompressed data. Ironically, if it had been zipped it would have been spotted.
UPDATE: Maybe you are right.
Compression is precisely nothing to do with file integrity. Indeed ZIP files have cyclic redundancy check fields which provide integrity checks.
Compression is used all the time in business computing and if there was the slightest issue about the integrity of data compression it would not be used. In fact the whole area is covered by information theory, and of all the objections to (loss-less) compression, data integrity is probably the least.
Strawman
Ahhhhh… I can always count on more bitching and moaning garbage to go along with my morning brew while scanning SAR and rest of the photography gear community! How about this… none of these annoying reviewers actually know with certainty what kind of format is coming out way, so please, STFU until you actually get your hands on A7Sii and see first hand. Took them less than 24 hrs to find some new and MAJOR problem to complain about.
Ahhhhh… I can always count on more biiitching and moooaning garbage to go along with my morning brew while scanning SAR and rest of the photography gear community! How about this… none of these annoying reviewers actually know with certainty what kind of format is coming our way, so please, S.T.F.U. until you actually get your hands on A7Sii and see first hand. Took them less than 24 hrs to find some new and MAJOR problem to complain about.
I new yesterday- despite the 14Bit news- we’d still be hearing about this for a loooooong time.
Well for one, this will no doubt invigorate users to speak their minds about more great features that Sony could implement in their camera’s as well. ie, I myself would love to see a configurable half press(shutter) magnification window like Canon’s ML does for Sony mirrorless camera’s. Which would be a true joy for use MF lens users.
Likewise, it also look as though the power distribution and energy management (grip etc) could use some improvements.
I’d also love to see better bulb mode implementation catered to Landscape Astrophotography but, I accept that these are secondary to the initial cooked RAW scenario.
Exactly, lots of pointless debating about what the new raw format will be and what it means. Just wait till its released or Sony releases more info. Until then its all speculation.
Oh! Gentleman, you misunderstand Sony RAW.
Sony RAW is not “Lossless Compress”.
Sony 11-7 Bit compression is not recreate 100% accurate image. It is approximation compress.
So, Sony added 100% accurate RAW, using Uncompressed RAW format.
Sony does not have Lossless compress technology yet, so Sony must use big data Uncompressed format.
I prefer Lossless compress.
So am I right in saying that this will be true 14-bit uncompressed RAW?
If so, this is going to be total bliss for someone like myself, as I shoot landscape with heavy processing. – A+
I must admit, the whole discussion in this thread turns out to be a little silly. What matters is that we have a lossless RAW option.
Did you know that people have a right to expression and thoughts in internet forums? 🙂
Yep. Sorry.
But it’s still a little silly.
OK, I keep my mouth closed now. 😉
My comment was not intended to sound rude or tell you to shut up but more along the lines of remembering that people enjoy sharing their thoughts and opinions on matters. Which can often result in controversial statements.
That said, I’ve been following the A7R II cRAW dlema since it first began on SAR, and in spite of all the controversy, I’d say that there but only a select few who have proven to be genuine trolls on matters.
You have switched up Esstee, it was lossy compression and now it’s “file integrity”
It was always about file(RAW) integrity.
A concept that you apparently don’t understand based on your usage.
in·teg·ri·ty;
2. the state of being whole and undivided.
The part you don’t get is the fact that a file produced with a lossy compression algorithm can have 100% integrity, so can a compressed JPG file, or a losslessly compressed file, or a file produced using no compression. The term integrity simply means the file and it’s data is not damaged or corrupted. The context in which you are using this term is not valid. And frankly this repetitive incorrect usage is making you look like a dolt.
Nah.
dōlt – A person who is stupid and entirely tedious at the same time. Many times they are oblivious to their own mental incapacity.
HAHAHA!
Yes… they right to be stupid is God given… so many don’t have a choice.
You don’t have lossless RAW option. Just uncompressed RAW option.
Which is likely to be lossless by its very nature.
How does that make sense? The Sensor does not produce a lossy image data… the compression algorithm that is run in the processor throws away some of the image… that is what make it lossy… save the original data before compression and you have lossless uncompressed RAW…. better yet add or replace the lossy compression algorithm and you have lossless compressed… its actually simple… only issue is that on existing cameras Sony may not have space for both lossless and lossy compression algorithms on the firmware and so may be choosing to keep the compression they have and just add the ability to not compress at all…which would only require a minor change in the processing software to skip the compression logic and just save the image data….
>only issue is that on existing cameras Sony may not have space for both lossless and lossy compression algorithms
obviously you are not a software programmer
Hahahahaha….really? What makes you say that? Because that is exactly what I am…. But go ahead enlighten me you hacker you…..
yes – really!
I am software developer for embedded systems and I know very good that space for small algorithms is not the issue since 10 years. RAW-compression is a very small algorithm.
Maybe you are an oldschool software developer dealing with 8-bit processors in assembler 🙂
I don’t believe you are a software programmer. I claim you are lying.
Well that just hurts my feelings so much … Because I care tremendously what you think of me … 🙂
If bionz X can make jpegs in high-speed, it can compress RAW lossless with even less computing power.
11-7 is just a piece of code applied to the image data to make a lossy cRAW file from the uncompressed image data coming off the sensor… apply a different program that does not throw away data during compression and you get lossless Compressed Raw … files will be bigger… but still smaller that uncompressed image data… Lossless compressed and uncompressed both give you back exactly the same image… in the best of all worlds… Sony gives use all three options (like Nikon and I’m assuming Canon do)… but in the long run as long as I have a lossless option I’ll be happy…
>Sony does not have Lossless compress technology yet
But I have. I did download it for free 🙂
Do you think I should sell this to Sony for 1 Billion $?
Even lossy compression, if done right, can be amazing and have no visible effect on properly decompressed image. Sony ARW2 is just primitive and stupid.
Maybe proper lossy compression algorithms were patented before Sony got into the game?
I don’t care if it’s compressed or not, as long as it’s lossless.
http://blog.sony.com/press/sony-announces-addition-of-uncompressed-14-bit-raw-still-image-capture-for-new-%CE%B1-cameras/
If you read the actual press release it clearly states:
“New α7S II to Feature Selectable Compressed and Uncompressed 14 Bit RAW at Launch”
Which I take to mean you can choose compressed or uncompressed 14 Bit Raw.
They don’t say lossy or lossless, so I dont really get why people are getting worked up by it… Unless they’re already looking for the next thing to whinge about. Then again, I don’t short star trails pushed to the end of the slider in post, so I can live with Sony’s standard raw format (which has served me well for years)
I don’t care either way personally.
So long as the artifacts are gone when doing extreme shadow recovery and around highlights in night shots. I’ll be happy is a pig in ****
Yes, but there is NO 14 Bit RAW!
It’s a lie.
All they offer is 11-7 lossy compression.
So, we’ll be still having 11-7 lossy file, just uncompressed?
WTF Sony?!
Have you independently verified this or just more concern trolling on your part Dumbir.
http://3zgehi1uaxi23dphbrgqa50r6z.wpengine.netdna-cdn.com/wp-content/uploads/2015/09/Chris-Gampat-The-Phoblographer-Sony-A7s-Mk-II-first-impressions-product-images-7-of-11ISO-2001-40-sec-at-f-2.8-680×453.jpg
Excellent, now what are the other options?
http://www.thephoblographer.com/2015/09/15/first-impressions-sony-a7s-mk-ii/
Interesting, could you show there it allows you to choose 14 bit or 12 bit raw file?
You can’t have 11-7 uncompressed. The more ignorance, the more joy.
IIiah… the 11-7 compression is applied to what? What is it that is being compressed? Answer… a 16 bit image produced by sensor … 14bits data and 2bits overhead info… now rather than applying the 11-7 lossy algorithm they use now in the Boinz X processor, they could do what Nikon does with their 36mpix Sony sensor… leave the data uncompressed, or use a lossless compression algorithm or apply a lossy algorithm (11-7 in Sony’s case)… Then allow the user, eh..sorry photographer, to select which they prefer from the system…eh..sorry camera… menu and bingo….nirvana… hope this helped…. you obviously seem confused.
> a 16 bit image produced by sensor
No such thing, sorry. First of all, sensels themselves do not produce digital bits, it is analogue signal converted via ADC. Second, ADCs are controlled with a ramp, and the ramp is 13-bit at best currently. 11-7 compression is performed either in ADC (you can look up the ASC architecture if you wish to see how such things are implemented), or on a DSP.
Iliah is right; there is no 16-bit image from the sensor. It is an analog signal that needs to be converted to a digital value by the A/D converter. With the information available to us outsiders, we cannot be sure if this digital value is 14, 13, 12, or 11 bits. (We can tell from the data in the files that it is not less than 11, and if it was more than 14 Sony would almost certainly be trumpeting it in their marketing messages.)
At some point, the values are passed through a non-linear operation that reduces the digital value to 11 bits. The discarded values are at the high end of the range where the quantization error will make up a smaller percentage of the value. This is also where shot noise in the analog signal will be largest, so the discarded bits were mostly noise anyway. Without access to the full internal specs, we outsiders cannot be sure whether this non-linear compression happens inside the A/D converter or if it is done digitally by the Bionz processor. If it is done by the ADC hardware, there is no firmware change that can change that.
The final step converts from 11-bit digital values into blocks of 7-bit values. This step is definitely done digitally. Is this digital logic included in the ADC or in Bionz firmware? It is possible to do either, but the most likely place is in the Bionz firmware. If so, it would be easy for Sony to skip this step and just store the values straight out of the ADC.
Completely incorrect. The sensor readout is 14 bit, the lossy compression is then done using an algorithm by the image processor which is a seperate component and is driven via the camera’s firmware. Don’t apply the algorithm and you no longer have any loss of information.
> The sensor readout is 14 bit
Not in lossy mode.
> the lossy compression is then done using an algorithm by the image processor
Or directly on ADC.
Nope, Sony insists that the Bionz X processor outputs 14bit encoded images.
The actual processing even uses 16bit encoding, probably to avoid some data loss.
http://discover.store.sony.com/sony-technology-services-apps-NFC/img/editorial-layouts-e1/e1-6col1-imaging-chart04-desktop.png
“Nope” to what? Data coming from the sensor is biased by the black level (for example, for Sony A7r camera black level is 512 and this value is added to all the data) and is compressed through the tone curve equivalent to http://www.rawdigger.com/sites/www.rawdigger.com/files/Posterization/image10.png the compressed result contains 11-bit values.
Bionz being capable of dealing with 14-bit input is what makes the lossless 14-bit raw possible.
Nope to trolls like you
Are you saying you have a 14-bit raw file, ARW format?
And are you saying you waited for Sony’s new firmware to come out, got a file, tested it and they STILL have the problem ?!?
Or you continue with this unstoppable complaining without actually knowing what they did which is called trolling.
> >The sensor readout is 14 bit
> Not in lossy mode.
I also said: “Bionz being capable of dealing with 14-bit input is what makes the lossless 14-bit raw possible.” What are you missing?
How do you know they are using “lossy mode” when uncompressed RAW option is selected ?
Are you sure you know 100% of the Bionz X chip functionality?
How do you know that they are not doing the processing for the uncompressed RAWs completely in software as a worst case scenario?
There are so many information you are missing, but start blaming the company straight away. At the end of the day the engineers there built it, they should be able to figure out what is wanted of them and do it.
> How do you know they are using “lossy mode” when uncompressed RAW option is selected ?
Where did I say that??
The first two lines from your previous post
”
> >The sensor readout is 14 bit
> Not in lossy mode.”
You lost track. Full phrase I answered to was “The sensor readout is 14 bit, the lossy compression is then done using an algorithm by the image processor which is a seperate component and is driven via the camera’s firmware.” – and the answer was that in lossy mode there is no 14-bit readout.
And I said how do you know that they don’t have a mode that is not lossy ?!?
How do you know they can not read the sensor output directly 14bit and process it in software to produce uncompressed lossless 14bit RAWs ?!?
(you know they can change the firmware which means adding new software, there is a version of linux running there after all)
> I said how do you know that they don’t have a mode that is not lossy
What are you referring to, exactly? Are you that confused? I said “Bionz being capable of dealing with 14-bit input is what makes the lossless 14-bit raw possible.”
What I am saying is that you don’t have the full specification of the Exmor sensor, the Bionz X processor, a schematic of the board with these components and the source code of their firmware.
Yet you still continue to insist that someone is lying to you and trying to scre.w you over.
Have some f’ing faith in the engineers that built the camera to find a solution.
They have all the information and you don’t.
At least we’ve seen some great cameras made by Sony engineers so let me trust them that they can come out with a 14bit uncompressed RAW format.
Instead we keep listening to you repeating that they can’t without even waiting for the finished firmware to come out and this is called trolling.
All you are saying is that you are confused, not reading, nervous, and using foul language.
You do not know what I have, and what I do not have. Stop guessing.
> they can come out with a 14bit uncompressed RAW format
My words are: “Bionz being capable of dealing with 14-bit input is what makes the lossless 14-bit raw possible.” – what exactly are you trying to argue with? Your misreading? Your lack of comprehension? Good luck with that.
No the confused person is you. You first insisted the problem is coming from the sensor output, your words
“Data coming from the sensor is biased by the black level”
Then you started blaming the Bionz X input, your words
“Bionz being capable of dealing with 14-bit input”
So what is your current position now, are you blaming the sensor or the imaging processor for not having real 14bit image data here?
You know all the articles discussing the details of Sony’s raw compression and what the artifacts are and exactly why it’s a problem? They pretty uniformly link back to the same guy who did detailed analysis of the data stored in the raw files and what the numerical issues with that data were. I’ll argue that he’s pretty much the root of why Sony is finally doing this, by laying out careful analyses that others such as DPR could refer to, so that everyone could understand the nature and circumstances of the problem.
And you just badmouthed that guy, one Iliah Borg. If there’s one person you probably shouldn’t be arguing with on the encoding, that’s the one.
badmouthed? really?
So you’re saying the guy patiently waited for Sony to release the new firmware, made a few uncompressed RAWs that he tested, confirmed their RAW file pipeline still causes artefacts and then continues the unending complaining
or
just continued complaining without even waiting for the finalised firmware to come out, which is called trolling.
Sony were pretty good to respond so quickly, lets wait and see what their engineers have done.
Or ratheher because 16 bit is “standard” in IT these days and I doubt that the BionzX processor is custom built from the ground up. It may simply be better from a project efficiency point of view to go with a “2 bytes per value” setup.
True, good point
Well, that would just get them more negative press if the “symptoms” don’t improve. I doubt they would do that.
Eh?? the 11-7 is the compression algorithm they use now…before you can compress you have to start with an uncompressed file… which is 14 bit… for true uncompressed raw… the save the original image and don’t bother compressing it… and so….ergo you end up with a 14 bit, uncompressed file…. now the only question is … will there be true uncompressed, lossless compressed, and lossy compressed (i.e. the 11+7 approach they use today)…
> before you can compress you have to start with an uncompressed file
Not necessarily. You can compress (including seven-eleven) data streams, even at the analog-to-digital conversion stage.
Oh please….so take the data stream and save it without compressing it and ergo…uncompressed file….
If the hardware is designed to allow it, and if it is what you are willing to do, yes.
Point is, to get a compressed file it is not necessary to start with a non-compressed file. To get a lossy file, it is not necessary to start with a lossless file.
The 11+7 scheme has two steps:
Step 1: Compress an analog brightness value that has a 14-bit dynamic range into a digital value that uses only 11 bits. This first step might be done as part of the analog-to-digital conversion process. (Do a Google search for logarithmic ADC to see an example of how this works). If the compression occurs in the ADC there was never any 14-bit digital value to work with, and there is no possible firmware change that could change that.
Step 2: Compress blocks of the 11 bit-values into 7-bit values. This step is what is causing most of the visible artifacts that people have been complaining about.
It would be very simple for Sony to skip step 2 and just save the 11-bit values straight from the ADC. Doing so would only increase the file size by 37.5%.
You really did not think that through. How can you have 11+7 lossy compression….just uncompressed?
lossy vs lossless applies only to compressed files. if it’s uncompressed there is nothing to lose
Uncompressed files bring a huge stress in storage. Lossless compression is what they should provide. I hope you don’t have to wait untik mk 3s hit the market before having access to it.
If you try to repair or restore a compressed raw file, your chances to succeed are much smaller compared to repairing or “un-erasing” an uncompressed raw file.
Maybe, but StellarPhoenixMacDataRecorery works fine pulling data off a Raid 5 array for <$50, and does a stellar job on Sony ARW files too. Another manufactured area of non-concern, better off worrying about Global Climate Change drowning NY in our opinion. Like we pointed out yesterday, as soon as Sony release this option, you ladies WHO ACTUALLY SHOOT stuff other than test charts will quietly switch back to the original lossy ARW format after seeing no difference in 99.999999999% of real world imaging! Go pop a Prozac, take a drive and make some pictures would be our advice here……
Lossless compression is just more convenient to work with. Since it’s lossless, there is no need to recover any “loss” that doesn’t exist. Sony’s current compression is lossy, which is the problem. Compression is fine, lossy compression is not.
https://en.wikipedia.org/wiki/Lossless_compression
Chris, you seem to completely miss the subject here. First, I’m comparing lossless compressed to lossless uncompressed. Second, if you ever tried to work with a file damaged during a copy operation you would know that uncompressed files have more localized defects than uncompressed.
I don’t get your point. Lossless is lossless because you can fully recover the uncompressed file from compressed file. So they are the same other than file size. Dictionary compression is not an unreliable algorithm. Second, if you have problems coping your files, you need 1) a better card reader; 2) a better card. If you are super scared about losing files, get a dslr that has 2 card slots and allows you to backup while shooting.
> if you have problems coping your files, you need 1) a better card reader; 2) a better card.
When the issue hits, it is too late to think of a better equipment. Not to mention that even the best equipment can go south, and does go south. Unfortunately, it happens more often than one may think, and can be even a matter of an insurance claim.
Then it’s another reason why you want to buy a dslr, as suggested. You can have a SD card backing up all the raw stored in the primary card.
Dual card setup is better of course, but it happens that the damage is done right in the camera because of some glitch, and both cards have same damaged raw. My take is that having the choice is good, lossy compressed, visually lossless compressed, lossless compressed and uncompressed is a good set of choices. And of course before erasing a card one needs to look through the raw files (not embedded JPEGs, but raw files) to be sure the copy operation went without an issue. An alternative would be to use some robust replication process with some checksum verification after the copy, during the readback. It is a good niche for a small software developer company.
You are overly worried about this.
> You are overly worried about this.
I deal with people who got file corruption issues _daily_. Yes, I’m worried – but not about myself. Overly or not – sorry, but it is not something for other people to decide 😉
If you put too much stress on your card, it fails quicker.
I guess Sony should have wordsmithed it a little better 😀
being vague is what sony does
Sure about that? All this discussion sure does not hurt them. People will argue every inch of value this might have. That sort of reasoning is hard to get by usual commercials.
If your loss is particular noise in every pixel (as opposed to
statistical distribution of such noise), it does NOT cause any image quality
loss. An ideal RAW format would actually be lossy, throwing out the
garbage (with a proper RAW processor restoring the noise if needed with
the same statistical properties). RAW files would actually become
smaller as ISO rises, so both single-shot landscapers and sports
shooters would be happy.
No, noise reduction always robs you from sharpness, and any form of processing the camera performs just reduces image quality. Software on a PC always does much better job than any camera. No to mention post-processing performed by a camera is permanent and can never be improved, whereas RAW processors only get better over time. Lossless compression (like a ZIP) is the only thing a camera should ever do.
Note the word “ideal”, no NR works in that ideal way.
I am not talking about noise reduction. Just about loss of information about particular noise in each pixel (while preserving statistical noise distribution picturewide, so similar-looking noise could be restored).
I wondered this yesterday. My personal take is that either is better than lossy compressed, but if given the choice I’d hope for true lossless compressed.
Now Sony have listened on the lossy compression in stills, perhaps they can stop equally crippling video with 4:2:0 8 bit in 4K? We really need 4:2:2 10 bit for pro level work.
How much it will impact buffer and performance depends on if its truely uncompressed or lossless compressed 14-bit raw… I would think that given the doubling of size for true uncompressed most all performance parameters with be cut in half… 2-3 shots per second… half the shots to buffer before full and pretty slow preview and write to cards (since its already limited)… BUT this uncompressed processing is an option to the current compression … which for me is good 99.5% of time… so for those times when its required its FANTASTIC to have the option…Thank You Sony.
You are right. In most images, the existing compression has zero losses for the majority of pixels and minimal losses for most other. The place where it really falls apart is when there is a very bright pixel within 16 pixels of a dark area. Getting rid of compression will increase the size of the file but will provide improved image quality in only a few cases.
Why should it impact the fps performance? My a900 shoots 5 fps regardless if it’s cRAW or RAW.
Another interesting question to think about. My camera can record 12-Bit uncompressed and 14-Bit uncompressed raw, which is considered lossless? More importantly, which is the format you want?
Both.
Frankly I want neither. I am happy with 14 bit lossless compression. Shoot the 14 bit uncompressed for a while and it does nothing better rather than fulling my hard drive and CF cards quicker.
If the compression is truly lossless there will be no difference other than the size. The data of the expanded lossless file and the data of the uncompressed file should be a bit for bit duplicate.
The core of this problem is that Sony force people to use lossy compression. Lossy is the key-word rather than compression.
It could be possible that Sony’s only rolling out a totally uncompressed version of the file, since a lossless compression would involve developing another algorithm. My guess is that although r/w will slow considerably, pp will speed up, like how LR speeds up considerably processing a huge 200MB+ tiff file over the original ~30m Alpha RAW.
It wouldn’t be 200mb to begin with.
More along the lines of 80mb
I know, the tiffs were 200MB.
That’s true. However, as it turns out, a RAW file is more efficient than a TIFF file in terms of file size. ie, a RAW file has one color stored per pixel. After which, the demosaic process creates the color image. As opposed to TIFF, which works out to be three color values per pixel.
No wonder it’s so fast to process.
haha passive sony the next troll question will be why no touch screen… and this no longer can fix via fw 😛
They can offer a programme to send your camera and get your screen replaced with a touch one for an extra $100.
How about android connection 🙂
They already have a app implementation, they just need to open for anyone to develop and publish the Apps.
It is really weird that Sony got this major underrated feature but doesn’t explore all the possibilities that it can bring.
That’s a troll comment about touch screen, knowing that’s what in the future of camera. Hahaha
Dear Bob:
I’m dealing with electronics for about 50 years 😉 I really do not need to use Google to look up what types of ADCs are here.
A ramp-controlled ADC SONY are using are perfectly capable of having a uniform ramp, and it _is_ the matter of firmware. In other words, even when “the first step” is done on ADC, it does not preclude from having different implementation and full 14-bit output. “Step 2” is often implemented on digital CDS circuitry, so for the purpose of this discussion without having vendor implementation documentation step 2 may be more complicated to skip – but I’m sure it is doable.
I have only been doing electronics and embedded firmware professionally for 30 years, so I bow down to your greater experience, Illiah. 😉
You bring up an excellent point about the non-linear ramp ADC. It is likely that the break-points and ramp rates for the non-linear ramp are probably configurable by firmware. That would make it POSSIBLE to switch to a linear ramp that supports all 16384 of the possible 14-bit values. However, I do not think this is likely:
A) It would take 8x longer to complete the ramp conversion sequence. Right now the memory write path speed is the limiting factor, but if the ADC conversion was 8x slower it would probably become the bottleneck.
B) The ADC hardware might not even support 14 bit outputs. If it was designed as a non-linear converter intended to generate 11-bit gamma-compressed values, it is likely that they would not have wasted the extra silicon to support 14 bits if they never intended to use all of them.
C) There is really no benefit (aside from shutting up critics) to using full 14-bit linear conversion at the bright end of the range. As this paper (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html) points out, the photon shot noise is bigger than the conversion step size, so no USEFUL information is lost in the 11-bit gamma compression process.
I can assure you the path is full 14-bit, as to the conversion time – it depends on many factors, some are programmable, like timing. As to the noise per the source you quoted, you can find much more detailed papers published between ’94 and ’99, and one of the interesting results is that you can get an image painting noise over noise. As to the memory bandwidth, we will see when the production-grade firmware will be made available to the public.