Migration v1.4 to v3.1.12 efforts

BBloke

Here's a new thread on my migration attempts. Tried all manner of things at the moment and I think I've just seen it work!!!!!!!

Marvellous. Only a few hopes to bounce through. So this is where I'm at and hopefully in the sequence I did it.

Made sure the server was using a utf8 collation/character set!
Change every table to utf8mb4 and utf8mb4_general_ci.
Change every table to InnoDB engine.
Altered the Database schema to utf8mb4 and utf8mb4_general_ci
Change database_connection_parameter in config.inc.php to utf8mb4
Ran my migration script.

Checked the new config file for v3 to ensure it had what I think it needed:

    'charset' => env('DB_CHARSET', 'utf8mb4'), 
    'collation' => env('DB_COLLATION', 'utf8mb4_general_ci'),
    'prefix' => env('DB_PREFIX', 'modx_'),
    'method' => env('DB_METHOD', 'SET CHARACTER SET'), 
    'strict' => env('DB_STRICT', false),
    'engine' => env('DB_ENGINE', 'innodb'),

For example.

Ran /install. Went through each step
I had a MGR_DIR message display on the last page but continued on to the log in screen.
Signed in!

Some of the above wsa derived from checking what a new installation did! Mainly changes to the collate, character set and DB engine on the new tables. So I did what I could to match it.

I've gone back to using the update.php script as I think from here it should be the safer bet.

I'll post up my mieage but I feel close!

user451

well done @BarnsleyBloke ! you are like Shackleton now 🙂 .... virgin snow and no one around 🙂 . mighty work.

fourroses666

Nice you have it working BBloke.

Out of curiosity, didn't the migrate script in Extras work or is it different?

BBloke

Well, I tried it and it failed but I don't think it failed because of the core code.

Sometimes I need to break things down for myself to understand what's going on in order to fix. It's sometimes a very long winding road.

I'm at the stage where I have my site code ready to drop in for testing.
I have the database and tables all set how I think it needs and I think I've gotten passed the COLLATION error I've been seeing.

So, I'm back to using the extras script.

I'm hoping my next attempt will be my last. Success is just around the corner.

I've gotta say that script is brilliant and very eloquent in using the new v3 core to migrate and the working with tables is genius. This is one of the reason why I thought about making the transition a module within evo. So at least the old code is doing the bits needed and then you finish it with /install. Sadly I hit the same brick wall at that part. (Nice to know exactly what the script does though!).

fourroses666

BarnsleyBloke Somehow I was always able to update sites with the migrate script. I did go to EVO 2 and then 3 for most of the sites. Not all went without issues but most of them where PHP version issues or database collation. Didn't have any webusers. Sometimes I had to run the installer over the site again. Anyway nice to have a new script to update v1 to v3.

BBloke

I have run the migration on v1.4.20 to v3.1.12 without issues and completed the installation process. I haven't gone into the manager to see if there is anything amiss but I have no errors.

I previous updated from v1.4.12 to v3.0.6 (I think) using the migration script. It worked without flaw but I think something more "core" has changed.

When I started this journey I duplicated our work site and DB and tried to run the update script thinking it would work. It failed at the COLLATION error. A fresh install sets its preferred parameters for the schema and tables whereas an old installation would be when standards and settings were different. I think my work site was done in 2007.

I've had to tweak MariaDB config to get the character and collation on the server (although I'm not sure it's required).
I've changed all the tables and schema.
I think that's what has proved to be my solution.

YES - The extras migration script has allowed me to complete the installation and sign in to the manager *

Despite having an error regarding a plugin and the web_users table.

SUCCESS!

BBloke

Something strange though.

Despite plugins being disabled I still got an error from the plugin code!

The site version number doesn't update until after the config has been saved!

BBloke

To finish off the installation I had to goto the Configuration page and save the settings so the version updated so any of my plugins that had version dependant code could run correctly.

I then updated the custom/composer.json to include Pathologics ModxAPI code, update Doclister and Formlister snippets and add the model in to the login Formlister code. A pretty small thing but something that needs to be done for it all to work.

BBloke

Just one thing to remember (I forgot but I'm sure I've posted about it before).

After successfully migrating over if you do have users (old web users) you need to remap your resource users/document groups for the web side in order for permissions on the frontend to work!

BBloke

Revisiting this and I'm surprised how far I have gotten.
There still some major things to address but I have it migrating and then install the latest version.
Sadly the if statements in snippets etc for the different versions doesn't work as I would have hoped it would.

The website version number only changes after successfully saving the configuration page post installation.

I'm currently getting an error trying to use the old user attributes table name!

So very very close!

BBloke

So Just posting here to help my adled mind!

My migration script works (this is the abridged version) but when I sign in to the manager I get the "web_user_attributes" table issue.

I'm not quite sure where this is bouncing from but here's a quirk. I deleted all my plugins as they run even if they are disabled!

Still get the issue.

I'm assuming that the latest master v3.1.13 is causing issues because when I copy an existing core folder to this installation things start to work better, i.e. no web_user_attribute table issue.

Going to try and knuckle it down by getting the previous v3.1.12 code.

fourroses666

BarnsleyBloke
If you see the error message:
Base table or view not found: 1146 Table 'xxx_xxx.modx_web_user_attributes' doesn't exist you need to deactivate plugin UserHelper!

Source: https://forum.evo.im/d/49-how-to-upgrade-evo-2-to-evo-3

BBloke

Turns out.

Deleting the plugins
Manually clearing the storage out
Works!

Just trying again!

BBloke

There are caveats to all of this but it would be great if I had a sample site which I haven't had any involvement with to really test it.

I know I've changed my database character set and tables to UTF8 (which I feel is a must!) to make it all work.

I've tested this on php 7.4 and 8.0. Whilst 8 has some warnings it does appear to work.

BBloke

RIGHT. I think I have it sorted out.

Make sure you create a backup of your files and database before taking this on.

I found that changing the character set for the DATABASE and TABLES was well worth it.

My current database is utf8_general_ci / utf8
My tables were also converted to utf8_general_ci

Install the module
Run the module evomigrate

This migrates the users and downloads the latest code.

Run the install for the new version.

Problem: web_user_attributes missing.
Reason: this is probably due to plugins.
Workaround:

disable all your plugins via database

Reload the manager page. It should bring you to the configuration page so you can save the new install config. Ensure you have your desired language selected.

Everything should be ok now.

Reload your plugins as needed.

BBloke

So the manager config saves but I'm not able to see the site working.

I think I have to update DocLister and FormLister which I will do tomorrow.

There may also may an issue with legacy PHX / and or friendly URLs!

More to Follow!

BBloke

Need to revisit and pay more attention to Permissions

BBloke

All sorted.

Site converted. Permissions working. Obviously some things to change regarding doclister and formlister but I think it's all working for the manager side.

It doesn't work when using PHP 8 (the migration) but I will look at that later. For now in version PHP 7.4 it's all good. You can change the PHP version once completed and the site will continue to work.

BBloke

I had it disabled but I had DashboardWebUser in there too so I opted to disable them all just in case.
I also felt I had some issues with getting past the error with plugins disabled but that could just be a victim of circumstance and/or caching!

Next step is to test with PHP 8.

BBloke

Tested on PHP 8 and ironed out the kinks and corrected a couple of bugs.