bpo-43567: Improved generated code refresh on Windows

[zooba]

Generated files are now refreshed automatically (based on last modified timestamp, which Git preserves) in Windows builds, or can be manually refreshed by calling "build.bat --regen".

https://bugs.python.org/issue43567

[gvanrossum]

Can you also add the lines needed to regenerate opcode.h and other files that must ge regenerated when opcode.py changes? See "make regen-opcode regen-opcode-targets". Also, frozen imports (basically everything done by "make regen-all").

[zooba]

Opcodes are regenerated in there already, there's just no command line option. Is there a need for one? It all regens pretty quick, and not at all if the files haven't been touched.

Frozen imports still have to be generated by the built CPython, so they're still in the _freeze_importlib project.

[gvanrossum]

Opcodes are regenerated in there already, there's just no command line option. Is there a need for one? It all regens pretty quick, and not at all if the files haven't been touched.

I tested this as follows: add a new opcode to Lib/opcode.py, add a case for it in Python/ceval.c, run PCbuild\build.bat. This gives an error in ceval.c, and opcode.h is not regenerated.

What am I missing?

Frozen imports still have to be generated by the built CPython, so they're still in the _freeze_importlib project.

Okay, that seems to be working. The workflow seems to be as follows:

• Change something that requires updating frozen importlib files
• PCbuild\build.bat
• This gives an error "importlib_external.h updated. You will need to rebuild pythoncore to se
  e the changes."
• PCbuild\build.bat
• "Build succeeded."

I'm guessing that I was used to the Mac/Linux workflow where this is part of make regen-all. But I don't see opcode.h being regenerated.

[gvanrossum]

There's also opcode_targets.h, which is generated by this snippet in Makefile.pre.in:

regen-opcode-targets:
	# Regenerate Python/opcode_targets.h from Lib/opcode.py
	# using Python/makeopcodetargets.py
	$(PYTHON_FOR_REGEN) $(srcdir)/Python/makeopcodetargets.py \
		$(srcdir)/Python/opcode_targets.h.new
	$(UPDATE_FILE) $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/opcode_targets.h.new

[gvanrossum]

Yes! This all works now.

I still find this block of errors hard to read:

C:\Users\gvanrossum\cpython\PCbuild\_freeze_importlib.vcxproj(146,5): error : importlib.h, importlib_external.h, import
lib_zipimport.h updated. You will need to rebuild pythoncore to see the changes.
C:\Users\gvanrossum\cpython\PCbuild\_freeze_importlib.vcxproj(146,5): error :
C:\Users\gvanrossum\cpython\PCbuild\_freeze_importlib.vcxproj(146,5): error : If you are not developing on Windows but
you see this error on a continuous integration build, you need to run 'make regen-all' and commit any changes.
    1 Warning(s)
    1 Error(s)

It's a lot of text that basically means "please re-run PCbuild\build.bat again," but I understand that the error may be produced in contexts where that phrasing wouldn't work either.

We should probably add a few words to the devguide section for Windows once this lands.

[zooba]

Yes, we expanded that text because it was confusing people when it was shorter. Tweaking it to just say "rebuild everything" rather than "pythoncore" probably makes sense. I'll do that before merging.

Unfortunately, I don't think I can get rid of the full path of the project before the message...

[gvanrossum]

Do we really need the part @(_Updated->'%(Filename)%(Extension)',', ') updated.? That makes the first line of the error twice as long. And I'd change "You will need to" into "Please".

[zooba]

I'll move the list of files into a regular message, so it gets logged, but doesn't make the important message noisy.

[gvanrossum]

Thanks!