Using P4Transfer to Migrate Your Perforce Server

Truitt Savell

Migrating Perforce Servers While Preserving History with P4Transfer

One day Bob the build engineer was tasked with migrating his team’s Perforce server from the computer living under some dev’s desk to a new and shiny cloud hosted server. Bob thinks to himself, “This will be easy! I can just download each of the streams from the current server and commit them to the new server.” “Wait!” the Release Manager yells across the office, “I need the commit history! You can’t lose that!” Bob’s smile falls to a grimace. “How can I migrate to a new Perforce server and preserve history?” Bob should use P4Transfer! With P4Transfer, a touch of configuration, and some time you can migrate your current Perforce server to a new one and preserve every commit across every Stream and Depot on your current server.

What is P4Transfer?

P4Transfer is a utility maintained by Perforce designed to assist in moving files from one server (the Source Server) to another (the Target Server) while maintaining history. You can find P4Transfer on GitHub and its documentation here.

P4Transfer works by using your local machine as an intermediary. It checks out each change from the Source Server into a local workspace, which it creates and manages, and then submits that change to the Target Server. This process effectively rebuilds the history from the original server. Keep in mind that the migration time will increase with the number of commits in your Source Perforce server as well as the overall size of the depot.

How We Used P4Transfer to Migrate to a New Server Provider While Maintaining History

Here's a breakdown of the steps we took to migrate using P4Transfer:

  1. Setup:
    • Checkout the P4Transfer Repo: Start by cloning the P4Transfer repository from GitHub.
    • Install Python 3: Ensure you have Python 3 installed on your system.
    • Install P4Python: Use pip install p4python to install the P4Python library.
    • Install Requirements: Install any other dependencies using pip install requirements.txt.
  2. Generate Configuration YAML: Create a sample configuration file using the command python3 P4Transfer.py --sample-config > transfer.yaml.
    • The transfer.yaml config file is used to define all the parameters needed to facilitate the transfer of commits. It contains the information for the P4 Servers and the Stream mapping for the Source and Target servers.
  3. Modify the transfer.yaml Config File: Carefully edit the transfer.yaml file following the comments provided. Pay attention to these key points:
    • If your Perforce servers do not have Character Encoding enabled, set p4charset: none for both servers in the config. If you are using Unicode or another type of Character Encoding, set this variable appropriately for your server. P4 Charset docs: Configure clients for Unicode.
    • Define the views of streams to be transferred. In our case we were transferring the entire depot and its streams, so we used the stream_views: section.
      • When using stream_views:, comment out the views: section as they cannot be used together.
      • Example stream_views: configuration: 
        - src: //src_depot/folder1/folder2
  targ: //trgt_depot/folder1/folder2
  type: mainline
  parent: ''
      • In our use case, we wanted the new server to look exactly like the previous server. So we simply set the src and targ sub folders to match each other. You have control here to map things differently to your new server if you so choose.
      • You will need an entry as above for each of your streams that you want to transfer
      • You can choose to set up the Streams in the new server beforehand, which we chose to do, or allow P4Transfer to set the Streams up for you.
  4. Start the Transfer: Once configured, run python3 P4Transfer.py -c transfer.yaml to begin the migration.
    • Transfer Time: The transfer time depends on the amount of data and the number of commits. Our migration of 100GB of data and 450 commits took about 15 hours over a gigabit internet connection.
  5. Resuming Transfers: If an error occurs or the connection is lost, you can restart the script. It will resume from the last successful commit, tracked by the counter_name: in the YAML config. This is extremely useful as it will ensure work is not duplicated and allows for you to fix local problems and run the script again without losing hours of machine time.

Possible issues you may encounter

This overview of using P4Transfer represents a perfect scenario of the use case. During our usage a fair amount of investigation was needed to come to these relatively simple steps. During your usage of the tool you might find that it does not behave exactly how you would expect. One of the troubles we faced was that we had files checked into our Source server that should have been ignored by our .p4ignore config. During the transfer when these ignored files were being transferred we saw errors as they were ignored during the commit to the Target server. Our solution was to directly modify our local checkout of P4Transfer to force commit files and disregard the .p4ignore config. The benefit of P4Transfer being an open source tool is that we were able to make these in the moment changes to fit our needs.

Conclusion

Congratulations! With a little bit of configuration and setup, and an indeterminate amount of time spent by your computer overnight, you were able to migrate that old Perforce server into its new home on a shiny and fast machine in the cloud! And your Release Manager is happy too with their preserved commit history! With everyone happy you can now go take a well earned bre… wait… nevermind builds are broken, gotta fix that now.

Thank you for reading and I hope you found this guide to be a helpful shortcut to learning about P4Transfer!