imported entities not displaying

[michielbdejong]

First of all, thanks for creating the export format with the entities.sql file, that's a great step!

I imported the entities table for a customer, but the imported posts are not displaying. I updated their URLs and owner URLs using search-and-replace (I had to do an import-and-dump first to get the hexadecimal blobs to turn into text which I can search-and-replace in with a text editor), but that was not helping. I analyzed the MySQL queries, and found out that it is trying to go to metadata for the access: 'PUBLIC' part. Looking at the code, maybe I should just create one entry in the metadata table for each imported entity?

[michielbdejong]

It looks like apart from the access: 'PUBLIC' metadata entry, a second metadata entry is needed for the slug.

Should I manually create these two entries for each entity I import? Or would it be better to import both the entities and the metadata tables?

[michielbdejong]

Should I try to write a PHP script that extracts the JSON data from the entities table and converts it to entries in the metadata table for the missing access and slug values? That might be an easy and quick workaround, I can publish it in a gist for others to reuse.

[michielbdejong]

@benwerd I got quite far with this, but now I'm still stuck. I worked out I need to create at least the following metadata records for each entity:

access: 'PUBLIC'
slug: the last part of the URL (without the '/201*/' year prefix)
object: (the full URL)
verb:"post"
owner: (the owner URL)
uuid: (the full URL)
_id: (the _id in the entities table)

It's now already displaying the 'Older' button on the front page, so the page is aware that the imported posts exist, but it's still not displaying the imported posts themselves.

Which fields am I missing? Is there an easier way to make this work?

[michielbdejong]

Ah, I found the problem. What's needed is:

• the Status entity, with metadata access: 'PUBLIC' and slug
• the Activity entity, with metadata verb 'post', and object === the uuid of the Status entity.

Looks like for the imported Status entities, not only the metadata, but also the accompanying Activity entity is missing and /its/ metadata are missing.

[michielbdejong]

OK, I ended up using a variety of methods :)

First in bash: echo select contents from entities | mysql known > metadata.txt

Then in vim:

• :%s/\\\\/\\/g
• :%s/\\"/\\\\\\"/g
• :%s/http.....oldname.withknown.com/https:\\\/\\\/newname.com/g

Then in nodejs:

var entities = require('fs').readFileSync('metadata.txt').toString().split('\n');

for (var i=1; i<entities.length; i++) { // skip the first line because it's the header of the mysql output
  if (entities[i].length) {
    var obj = JSON.parse(entities[i]);
    for (var key in obj) {
      console.log('INSERT INTO `metadata` (`entity`, `collection`, `name`, `value`) VALUES ("'
          + obj.uuid + '", "'
          + 'entities' + '", "'
          + key + '", "'
          + obj[key] + '");');
    }
  }
}

(save that as generateMetadata.js and then run that as node generateMetadata | mysql known)

This seems to at least generate all the necessary metadata entries.

[michielbdejong]

Do also take care to also run :%s/http.......oldname.withknown.com/https:\\\/\\\/newname.com/g (with triple backslashes in that case) on the 'exported_data' file containing the sql statements for adding the entities before importing that (if it's not in ASCII then you can convert it by importing and mysqldump-ing once), then this import method works.

@pierreozoux @timmmmyboy you may find these commands helpful when doing an import from a customer who came from *.withknown.com or elsewhere. Have you done any successful migrations yet? I just completed my first and second one! :)

[timmmmyboy]

Thanks @michielbdejong! Haven't had anyone need a migration of Known yet but this thread will be very useful when that time comes.

[pierreozoux]

Yes, thanks! No people yet, but yeah, it will be really helpful!

[davidmead]

Thanks @michielbdejong for this. I'm running into the same issue trying to import my earlier posts from a hosted davidmead.withknown.com to self-hosted blog.davidjohnmead.com.

The entities.sql imported fine and merged with my current posts. But apart from that, nothing. Now I see there's a number of hoops I have to jump through to get it working :-(

Where did you place the folders from your old Known blog? Under 'Uploads' in a new sub-folder, or did you merge them with the current folder in there?

[michielbdejong]

@davidmead yeah, importing the static uploads files was the easy part in my case, that worked as expected for me. Just merge uploads with uploads, and run chown -R and chmod -R to restore folder ownerships and perms. If images in imported posts don't display, check the apache error logs (or dtrace the process) to see which on-server file path it's looking at.

[benwerd]

Guys, I've just come back to see this thread, and I apologize for the SQL omission. We'll fix that - you definitely shouldn't need to write a script to import / export!

I meantioned to @davidmead on Twitter that we're working on an easier mechanism. I think this is crucial. Really nobody should have to muck around with SQL, either.

[michielbdejong]

@benwerd++ \o/

[benwerd]

Here's how I want it to work:

• User clicks "import content" in their new Known site
• User is prompted for their old site URL, clicks to connect
• User signs into old site
• Content is imported automatically

All thoughts on this appreciated.

[timmmmyboy]

Yeah turns out I now have one of these on my plate too and wasn't looking forward to having to do some fancy scripting in Node :). A SQL file would be fine for now, more than used to that and we can do a find/replace on links. @benwerd being able to import from an existing install would be awesome! I don't know how well that deals with mapped domains though where the old site and new site might be at the same URL as part of a migration. That can happen if someone is moving to or from Withknown.com and taking their domain with them.

[benwerd]

Yeah, it wouldn't work in that case. If you have access to the entities and metadata tables, @timmmmyboy, you can just move them across. Otherwise, I'll try and take a peek at the export format later today.

[davidmead]

Thanks @michielbdejong, but I when I read

run chown -R and chmod -R to restore

you've lost me. Last time I "knew" what to do with a command line was with my ZX Spectrum ;-) But there's guys I work with who can help me with that.

I've grabbed your post into Evernote and may try it later this week. I'll ping you if it goes horribly wrong.

[davidmead]

@benwerd I think that's exactly how I'd expect it to work. As a user I shouldn't have to see or do anything with the wiring behind the boards, so to speak.

It should be stated that only other Known sites can be imported this way, unless you're planning on doing that for WP too (fingers crossed. I have one of those too).

Export should a separate page/tab and it should be clear what you'll get from that.

[davidmead]

Also @benwerd, we have to think about how it'll notify search engines that content has moved from one blog to another.

After importing, would you offer closing down the *.withknown.com site? If so could you automatically implement a 302 redirect in an .htaccess somewhere?

[michielbdejong]

closing down the *.withknown.com site [...] implement a 302 redirect

cool idea! Also relevant for people who migrate to KnownPro, probably.

[pierreozoux]

@davidmead about wordpress, FYI you can follow conversation here.

[davidmead]

Thanks for the link @pierreozoux

[mapkyca]

@davidmead Actually, I suggested this the earlier today (https://mapkyca.com/2015/benwerd-withknown-they-should-get-an-option-to-set-everything) in response to a complain about the blogger export.

Definitely wise to present the option, although I would suggest that you use a 301 rather than a 302, so that search engines update themselves.

[michielbdejong]

This issue is no longer relevant, because now when you click '(re-)generate my archive' and then download it, the 'exported_data' file inside the zip is a .sql file containing both the entities and the metadata.