BUG: Using compress_identical_objects on transformed content duplicates differing content

[danio]

compress_identical_objects() can result in lost content, if page.add_transformation is used.

Test that creates bad output:

from pypdf import PdfReader, PdfWriter, Transformation

def test_transform_compress_identical_objects():
    reader = PdfReader(RESOURCE_ROOT / "two-different-pages.pdf")
    writer = PdfWriter()

    for page in reader.pages:
        op = Transformation().scale(sx=0.8, sy=0.8)
        page.add_transformation(op)
        writer.add_page(page)
    writer.compress_identical_objects()
    with open("result.pdf", "wb") as fp:
        writer.write(fp)

Input file:
two-different-pages.pdf

This contains "1" on page 1 and "2" on page 2

Result before fix:
result-before.pdf
This contains "1" on page 1 and "1" on page 2

Result after fix:
result-withfix.pdf
This contains "1" on page 1 and "2" on page 2

The issue is around the EncodedStreamObject which gets converted into a ContentStream after the transformation. BothContentStream objects on each page have the same obj.hash_value(), even though they have the same contents. Printing the _data value out for the contentstreams showed it was empty after the transformation.

I tried creating a simple test to reproduce just the hashcode calculation for the content streams but couldn't get it to work, I'm not sure quite what is happening inside page.add_transformation to create the content stream with no data yet calculated. This is what I tried (inspired by test_contentstream_arrayobject_containing_nullobject):

stream_object = DecodedStreamObject()
stream_object.set_data(b"Hello World!")
content_stream = ContentStream(stream=stream_object, pdf=None)
writer = PdfWriter()
cs2 = content_stream.clone(writer) # I also tried .replicate(), with the same result
print(cs2.hash_value()) 
print(cs2._data) # this was set correctly, I was hoping it would also show the issue

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.54%. Comparing base (a548ca1) to head (7567944).
Report is 74 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3197   +/-   ##
=======================================
  Coverage   96.54%   96.54%           
=======================================
  Files          53       53           
  Lines        8935     8935           
  Branches     1642     1642           
=======================================
  Hits         8626     8626           
  Misses        186      186           
  Partials      123      123           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[stefan6419846]

Thanks for the report and PR. Could you please add a corresponding test as well?

The function provided by you should be rather easy to transform into an integration test if you run page.extract_text().

As for why _data is not populated after the transformation: The content stream can either be an unparsed string or a parsed list of operations.

pypdf/pypdf/_page.py

Lines 1003 to 1018 in a548ca1

	@staticmethod
	def _add_transformation_matrix(
	contents: Any,
	pdf: Optional[PdfCommonDocProtocol],
	ctm: CompressedTransformationMatrix,
	) -> ContentStream:
	"""Add transformation matrix at the beginning of the given contents stream."""
	contents = ContentStream(contents, pdf)
	contents.operations.insert(
	0,
	[
	[FloatObject(x) for x in ctm],
	b"cm",
	],
	)
	return contents

is responsible for the actual transformation and operates on the operations, thus _data is not set unless you call get_data() explicitly. (contents is page.get_contents() in this case.)

[danio]

@stefan6419846 That test would be nice, but I can't see how I can extract the text from the PdfWriter.
When I try writer.pages[0].extract_text() I just get empty data.
Would the test have to write to file, then open it? (I'm thinking it should work with a BytesIO so no need to actually write to disk)

[danio]

@stefan6419846 I've added a test that uses BytesIO to write/read the resulting PDF, let me know if there's a better way and I'll change it

[stefan6419846]

The current approach looks fine for me. I have added a comment about a small code style change we should add; afterwards, I see no problem with getting this merged.

[stefan6419846]

Thanks.