Maureen Clements: Great.
Maureen Clements: Well Thank you everyone, I want to say, my name is Maureen Clements. I'm the new associate director for SUNY library services.
Maureen Clements: i'm in my second month so i've only met a handful of SUNY people, so far, but i'm very much looking forward to getting to know everyone and working forward together i'm joined today by my colleague, Michelle Eichelberger who many of you already know.
Maureen Clements: And, but before we get started, I just wanted to give you a very high level what we're going to be talking about today.
Maureen Clements: We recently found an issue with some data syncing between ALMA and oclc. It's an intermittent issue and so
Maureen Clements: that's why it was a little bit difficult to find and what we're going to do today is we're going to review what that issue is.
Maureen Clements: we're going to discuss how we intend to sync the data and what our plan is going forward. We're going to provide you with some information about doing this yourself and then we'll have time for a q&a session at the end.
Maureen Clements: So i'm going to actually just turn it right over to Michelle so she can get started great Thank you.
Michelle Eichelberger: Great thanks so much Maureen hi everyone, this is Michelle.
Michelle Eichelberger: Eichelberger Thank you so much for joining us today to talk about this issue to talk about our next steps, and what we plan to do about it for SUNY.
Michelle Eichelberger: So as Maureen mentioned what's happening is an intermittent problem and changes to your records and almost should be publishing to oclc we have the publishing profile setup.
Michelle Eichelberger: And they're being regularly sent out occasionally they do fail we'll talk about that in a minute, but what's happening is that the data doesn't always arrive at oclc, even though the publishing job in ALMA says that it succeed.
Michelle Eichelberger: trouble.
Michelle Eichelberger: Okay So how do we discover this problem? In early April, we received a campus report of Titles that had been deleted and Alma but weren't showing as deleted in oclc in this campus was publishing weekly So there you know, there was a possibility that there was a lag.
Michelle Eichelberger: But we kept an eye on it and those deletes never did show up in oclc so you know to troubleshoot this, this is actually a really good learning experience for me.
Michelle Eichelberger: You know, taking on some of the nz coordinator roles, so it was good to go through this and just make sure I understood how the publishing profiles were set up.
Michelle Eichelberger: How they should be behaving So what we did to troubleshoot this issue was we checked the oclc SFTP settings they looked fine.
Michelle Eichelberger: We looked at the data sync publishing profile settings for this campus they looked fine we compared it with other campuses it looked exactly the same, it was all set up correctly.
Michelle Eichelberger: We did look at the Alma job history to look for failures, because, as I mentioned, occasionally, you will have legitimate failures and then the data will get recent it's not a big deal it happens all the time.
Michelle Eichelberger: And then we also checked worldshare to see if the files had actually arrived from Alma and what we found is that that was not always happening.
Michelle Eichelberger: So when we did a comparison of successful jobs and Alma with the files received in world share, you can see a list of all the files that show up on your world share site.
Michelle Eichelberger: We found that there were actually missing files on the oclc side, and so what that meant was that, even though the job report might say status completed successfully if that was not always accurate, which was disturbing to find out.
Michelle Eichelberger: You know, as I mentioned ftp does fail fairly regularly and, if you look at the events report, for your jobs, you will sometimes see these error messages you know job failed to connect this happens all the time ftp is not.
Michelle Eichelberger: The most stable delivery method, it is sometimes the only way, you can deliver data.
Michelle Eichelberger: And both oclc and ExLibris are aware of these issues, and what is supposed to happen is the job is supposed to continue to try to connect.
Michelle Eichelberger: And you know, sometimes it will connect and then the job will go through, sometimes it tries too many times and it fails.
Michelle Eichelberger: And even that is okay, because if your job officially fails it saves all the data that was in there and It just seems that the next time so it's an you know if you see a failure that doesn't necessarily mean that the data did not end up where it was supposed to go.
Michelle Eichelberger: However, what we found is that sometimes the job got far enough along in the process where.
Michelle Eichelberger: On the ExLibris side it thought it was working so it got that successfully you know completed successfully identification, but right at the last minute before the file really got to oclc it failed so.
Michelle Eichelberger: Those failures were not being caught anywhere, so it was only because this campus person said hey why aren't our deletes showing up that we could even find out that this was a problem.
Michelle Eichelberger: The big problem with this is that, if that job officially fails, you know it says in the job listing that it failed it keeps all those records and it sends them, the next time.
Michelle Eichelberger: But if it thinks that worked that data is just gone, you know.
Michelle Eichelberger: When you delete things they get a little lowercase D up in the leader and that's what tells it to be sent to oclc if you've deleted, something that has gone from all I can't be recent again.
Michelle Eichelberger: That data isn't stored anywhere to be recent again if it thinks the job was successful so that's a big problem, and when we reported this to both oclc and ExLibris both sides said Oh, this is a known issue and it had been reported by institutions outside of SUNY.
Michelle Eichelberger: One of the really good things about this, I mean this is a.
Michelle Eichelberger: Not a great situation you know it's it's an intermittent thing it doesn't happen all the time, but on the bright side.
Michelle Eichelberger: ExLibris can see this happening, so you know i'm sure you're all familiar with times when you know that something's wrong.
Michelle Eichelberger: You can't prove it, that the vendor says, oh no it's everything's fine, but in this particular case, they can see it happening, which is great, and which will hopefully help get it resolved.
Michelle Eichelberger: So the challenge, then, is how do you tell which jobs really failed, you know, can you trust if it says it completed successfully that it really completed successfully there's no way to tell an online.
Michelle Eichelberger: Actually versus working on this and we're hoping by early fall, that we will have a better indication of when these things are really failing.
Michelle Eichelberger: You know if you wanted to do a full comparison of the files that were sent from Alma of with the files that are in WorldShare.
Michelle Eichelberger: Those world share files early kept for a limited amount of time, so you couldn't go back since go live in 2019 and compare what actually got to the right place.
Michelle Eichelberger: So that's a problem as well, the timing of when a file was sent from Alma and when it arrives at oclc isn't always a one to one match.
Michelle Eichelberger: There can be a lag in terms of when it was sent, and when it actually shows up on the world share so that can make it challenging to to kind of compare the two ends of the of the transaction.
Michelle Eichelberger: The other challenge is that it's really hard to tell what was in the file that was sent that didn't show up.
Michelle Eichelberger: The Alma report will tell you the number of records were in the file, but it doesn't tell you specifically which ones, and, as I mentioned earlier, the deleted bids only send once and after that fails they're usually gone from ALMA so you can't find them in Alma.
Michelle Eichelberger: There is a way to find them and we'll talk about that in a minute The other thing is, you know if your number of records says.
Michelle Eichelberger: I don't know 2500 records, not all of those are things that are held in oclc they may be, including electronic bids that you don't really.
Michelle Eichelberger: manage in oclc so there's no real easy way to say okay these 50 oclc records are the ones i'm wanting to deal with this is the job that failed i'm just going to reset the job that's not possible, unfortunately.
Michelle Eichelberger: To take a breather.
Michelle Eichelberger: Okay, any will save the questions to the end, this is actually a fairly short presentation so we'll have plenty of time to discuss the situation.
Michelle Eichelberger: So what SUNY library services is planning to do is that we're going to continue to advocate for improvements to the publishing job you know, of course, we want this to be fixed, but this has been a long standing issue that we just became aware of the spring.
Michelle Eichelberger: it's not going to be fixed quickly or easily, but we are going to continue to advocate for a fix for this we're meeting with development teams at both oclc and ExLibris
Michelle Eichelberger: And we're also communicating our findings to other consortia who this impacts as well, and I did see somebody from cuny on the call so that's great we really.
Michelle Eichelberger: appreciate teamwork and trying to make this better for everyone, because it would be really great if these two vendors, could we could get it together and only have to catalog in ALMA, which is what we are doing currently and we thought it was working better than it was.
Michelle Eichelberger: As far as SUNY goes, we will use the.
Michelle Eichelberger: LSP and discovery headquarters a cue to communicate any status updates, you know if if they do fix it, of course, we will let you know out there, you know we will communicate through that mechanism to let you know where we are in the process of this project.
Michelle Eichelberger: So, as I mentioned that the job we're not looking at this being fixed next week, this has been a long standing problem you know it takes a while to get things through development on both sides of the equation.
Michelle Eichelberger: So the expectation is that suny library holdings will continue to be out of sync and, as we said, this is Internet and it's not like every job is failing so that you're probably going to be slightly off for a while.
Michelle Eichelberger: So there is a way to deal with this and we'll talk about you know who's going to be responsible for what.
Michelle Eichelberger: But the data sync workflow what you can do is you can find the deleted titles using analytics there's a life cycle.
Michelle Eichelberger: field in analytics and I do have more information about this later, but you can find the deleted titles in there, then you can batch delete them and connection and then run full publishing for all.
Michelle Eichelberger: The running the deletes first that ensures that if you deleted something and then re edit it later.
Michelle Eichelberger: You won't accidentally completely delete it, so the deletes go first than the full publishing and when I say full publishing.
Michelle Eichelberger: it's using the same publishing job that's already set up but currently everybody set to incremental to only deal with the things that have changed since the last time, the job ran.
Michelle Eichelberger: it's just an easy you know you switch in the publishing profile to full publishing that will send everything, and then they should be in sync.
Michelle Eichelberger: until the next problem happens Okay, so what we're proposing that will do is, we will run the initial data sync workflow to catch the missing changes since go live.
Michelle Eichelberger: And this is an opt in situation you know if you've been watching the HQ I did post a thing this morning about.
Michelle Eichelberger: EBSCO which we were thinking would be an opt out situation because everybody gets it and there's more information out there, but this particular project we're looking to make it an opt in.
Michelle Eichelberger: So that because we need you to let us know if you want us to do the first run at this and we need you to provide us with your connection account information.
Michelle Eichelberger: So that's why we need you to fill out this form, if you would like us to do it for you.
Michelle Eichelberger: For any campuses who we have support contracts, we will automatically just do this for you, and not only will we do the first run for you, but we'll do any continuing runs.
Michelle Eichelberger: Until if and when this is ever fixed and is working correctly, but for everybody else in SUNY, we will do it for you, if you let us know that we that you want us to do it for you.
Michelle Eichelberger: we've also created a training faq which has helped has all the details about you know how do you run an analytics report to get your deletes.
Michelle Eichelberger: What are the steps to running a batch deleting connection, how do I set up for publishing that's all in this faq.
Michelle Eichelberger: And Maureen will put that in the chat so, our goal is to finish this first pass this like initial sync up up till now, for the supported and opt in campuses by the end of the summer.
Michelle Eichelberger: And once you get the delete file it actually goes pretty quickly, so I think that's a reasonable goal.
Michelle Eichelberger: The.
Joan Cook: Only one on that old thing.
Michelle Eichelberger: Job i'm gonna make.
Michelle Eichelberger: Maureen, could you mute Joan please.
Michelle Eichelberger: Thanks so much cool um so we have done a few campuses already and the rate of failure her it has very, very widely.
Michelle Eichelberger: prescribed for one of the campuses it was only like 9% of things didn't really get sent for another campus it was more than half of the things didn't get sent so it's we don't have any.
Michelle Eichelberger: Good sense of you know what's causing some places to have higher rates of failure.
Michelle Eichelberger: It could have something to do with daily versus weekly oclc publishing if you're doing it daily those are smaller files, so if you lose one it's not going to impact your holdings as much.
Michelle Eichelberger: You know weekly publishing if you're doing a ton of weeding you know week to week that could really impact your publishing.
Michelle Eichelberger: You know, it could be related to the time the week when the servers are more busy than others, one of the challenges with these publishing jobs is that there.
Michelle Eichelberger: Everybody in Alma is running them at the same time, so that you know that, like the same time of day.
Michelle Eichelberger: If it's a once a week everybody's running it on the same day, once a week so that can be part of the problem we don't have any.
Michelle Eichelberger: specific information about what's causing the problem that is something that development is looking into both sides oclc and ExLibris.
Michelle Eichelberger: And if we find any reportable data, you know just maybe look at the drop, but if we find anything that we you know have recommendations about such as you know, maybe everybody should do weekly publishing or daily publishing excuse me, you know we will post about that in base camp.
Michelle Eichelberger: All right, that is our speedy presentation on this topic, and now we will open it up for questions so just let us know if you have any questions about what our plans are or anything else.
Maureen Clements: And if you do have questions you feel free to use the chat.
Maureen Clements: Or you can also unmute yourself.
Maureen Clements: Okay, so Michelle i'll read this question out to you and.
Michelle Eichelberger: I can see it in the chat great.
Michelle Eichelberger: If you want to read it, for others might not see it that's.
Maureen Clements: Okay, so if we have a record in the IZ for print title and we delete it from Alma is it theoretically supposed to be published for deletion from world cat.
Michelle Eichelberger: Yes, it is created yep.
Maureen Clements: Okay, does the full sync jobs and both deletions and additions or just additions.
Michelle Eichelberger: kristin it sounds just additions and that's why you need to grab your deletes from analytics and run those in batch the connection first.
Maureen Clements: Okay, so Carol has a question what's the timeframe for checking everything we've been doing a huge weeding project and assuming things have been running fine, but our files could be huge.
Michelle Eichelberger: So.
Michelle Eichelberger: Let me scroll up to see what campus you're from Carol.
Michelle Eichelberger: So the timeframe for checking everything if you mean in terms of what sls can do for you, if you opt in and we're going to do it for you we're hoping to have everything done by the end of summer.
Michelle Eichelberger: You know you have to run the batch delete in groups of 2500 titles, so you know if by huge mean a million titles that could take a wireless by huge you mean you've deleted, you know deleted 1000 things 10,000 things.
Michelle Eichelberger: that's certainly doable in that timeframe we're looking at, and if you follow up questions for that definitely let me know.
Maureen Clements: that's right yeah and and also Carol if you have other questions you can reach out, you know, through the sls ticket system to.
Maureen Clements: Okay, so another question is, if we are manually updating holdings and cat express do we need to worry about this.
Michelle Eichelberger: That you Doug, you're at ESF and my understanding is that you're not using publishing from Alma anymore, so your holdings may be out of date anyway.
Michelle Eichelberger: We can follow up with you later for the cat express Susan's on the call Susan do you have a better answer I just don't know that much about cat express.
Susan Perry: know that that is true, when you make the switch from full cataloging to cat express your holdings kind of get frozen in time so publishing profiles are not at play so everything for cat express libraries, has to be done manually.
Susan Perry: Does that answer the question and Doug we can you know we can talk via email to So if you are unsure about that.
Michelle Eichelberger: And Carol I see your note about that you're from Potsdam 25,000 yeah again, you know that's it's certainly doable to get that done at a reasonable amount of time and it, you know.
Michelle Eichelberger: We have instructions, if you want to do it yourself, or if you opt in we can do that for you, for the first pass and then you can take it on going forwards, as often as you want to run it but yeah that's 25,000 is not terrible.
Maureen Clements: Okay, so we have a question from Joseph Riggie.
Maureen Clements: We continue to do manual add deletes and connection and rely on data sync as a backup failsafe does it make sense to continue the manual add deletes at least until oclc actually fixes this.
Michelle Eichelberger: So that's really up to you, I mean if you guys have been doing this all along and don't really want to change your workflow I mean.
Michelle Eichelberger: seems like you're smart to be doing this because we thought we could rely on the data sync more than we actually could, but even if you know.
Michelle Eichelberger: Even if you just rely on the data sync we have this process for for catching problems so that's really a local decision, how you want to manage your workflow.
Maureen Clements: Okay, and then Alana from Geneseo just you know Shannon.
Maureen Clements: responded, but she wanted to ask for a little more clarification on the deletes in the additions this proposed fix handles the deletes only right? or how will the additions be handled.
Michelle Eichelberger: Alana, it's actually both so the the analytics you run the analytics you grab your deletes so you know you look at what you've deleted you run a batch delete in connection.
Michelle Eichelberger: And the client has to be done in the client and then the full publishing from Alma is what pushes the additions out so it handles both pieces of it.
Maureen Clements: And we have a question from Renee do you have the option of manually adding deleting from connection like Joe Riggie just described.
Michelle Eichelberger: It would be the way that you catalog before we got.
Michelle Eichelberger: All my so you know it would be what you were doing before we had a publishing profiles from Alma.
Michelle Eichelberger: But again, you know, this is what we're trying to avoid we're trying to avoid people having to do their cataloging in both systems, the beauty of Alma was supposed to be that you could do everything and Alma and it would magically keep in sync with oclc.
Michelle Eichelberger: And for the most part, this has been happening, I mean.
Michelle Eichelberger: It is an intermittent problem it's not that it's consistently failing all the time, and we do have a mechanism for doing and catch up so.
Michelle Eichelberger: You know it's really up to you, and I would not recommend starting to catalog in both places, particularly if you don't have that much volume it's really a lot of extra work that you don't really need but that's.
Michelle Eichelberger: that's what i'm thinking.
Maureen Clements: And there's a question here from Chris Wilson is, is there a job or analytics report that you can run to help us figure out how behind we might be.
Michelle Eichelberger: The report tells you what's been deleted and then, when you do the batch deletes in connection.
Michelle Eichelberger: If it's already been deleted it just it tells you that in the file it'll only delete the stuff that hasn't been deleted, there is no mechanism that we had we didn't have an easy way to compare the two.
Michelle Eichelberger: The two systems we did look at that At first it was actually very challenging to even get a good list of how many holdings, you had from oclc we.
Michelle Eichelberger: tried, the same campus we went into world share and that you can set up a collection and it should tell you how many things you own.
Michelle Eichelberger: We got different numbers for the same campus.
Michelle Eichelberger: In multiple.
Michelle Eichelberger: You know login so that was challenging so there's no good way to tell you how behind your might be.
Michelle Eichelberger: You know you're probably not that far behind, particularly if you're doing daily publishing but, really, we found that the best way to deal with this was just grabbed the deletes.
Michelle Eichelberger: run the batch in connection and that just caught anything that might have been this, but that is a good question I wish there was a better answer for that.
Maureen Clements: And I just want to want to hop on that part is that there are a lot of problems between oclc data itself and worldshare, and you know that when your holdings there they're not thinking properly so so I don't think oclc whatever would be able to provide us with a definitive.
Maureen Clements: list of what they have so okay so there's another question here himself so.
Maureen Clements: This is from Karina Ricker i'm confused about when the ALMA data is gone and when it is still accessible via analytics.
Michelle Eichelberger: I would say, this is a beautiful thing about all you know I could complain about all my day in and day out, but one of the really great things about Alma is that it remembers everything.
Michelle Eichelberger: So you know in ALEPH, a lot of times, people would keep their bid records once they deleted, you know the items or whatever, because they needed to know what they had.
Michelle Eichelberger: What they had deleted, they would mark it as suppressed you no longer need to do that and online, you can completely delete the bib from your Alma.
Michelle Eichelberger: And that information that you used to own that thing is always saved for you and analytics.
Michelle Eichelberger: and the report that we've created that you can use on your own again it's the one will be using to run these deletes What it does is it looks at the life cycle.
Michelle Eichelberger: And it identifies any bibs and their oclc numbers and the mms ID that you used to have that you can no longer look up in Alma but it's still knows that you had it and you get a whole list of everything you've ever had that you've got rid of it's really great actually.
Maureen Clements: And i'm I noticed that Kristen had her hand up and I want to make sure that we, we were able to answer your question, because I think you had asked earlier on.
Maureen Clements: No, I don't have anything okay great.
Maureen Clements: Alright, so this next question is from Brett Arno is there any discussion with ExLibris to try to improve the publishing profile report to provide more useful information like titles.
Michelle Eichelberger: And it looks like Shannon.
Michelle Eichelberger: said yes yeah we would we would love it if that was better.
Michelle Eichelberger: yeah I don't think that's you know we're really hoping that they can do a better job at least telling us what's really failed.
Michelle Eichelberger: As far as like fixing the whole structure of the reports I don't think that will happen soon, but i'm sure everybody who uses on that would really appreciate that yeah and Shannon does say actually versus working to make improvements.
Maureen Clements: Yes, yes, they are there they're definitely.
Maureen Clements: we've been meeting with them on this issue so.
Michelle Eichelberger: And Wendy, the report to identify the Info is on the faq that Maureen had posted earlier in here and we can grab that and put the link in the chat again.
Michelle Eichelberger: Great.
Maureen Clements: got a little buried
Maureen Clements: hold on one, second, and this is the, this is the opt in form or was this just the.
Michelle Eichelberger: This is the FAQ
Michelle Eichelberger: yep so it's got all the specifics what we did was we created the report in a shared folder so that everybody in SUNY can get to it, if you want to do it on your own, which is totally fine you know we're happy to do this first pass for you.
Michelle Eichelberger: But if you'd prefer to do it yourself we're completely fine with that.
Michelle Eichelberger: And you know if you run into any trouble with any of the information you put in there, you can submit a Libanswers ticket and we're happy to kind of tell you.
Michelle Eichelberger: why we set it up this way what why we did it that way, just let us know.
Maureen Clements: And what we'll do too, and we post the slides and video will also include the faq and the link there.
Maureen Clements: Any other questions or.
Maureen Clements: thoughts questions.
Shannon Pritting: I just wanted to you know, point out that it seems like some are considering, maybe duplicate cataloging and both oclc and then doing it again in in Alma.
Shannon Pritting: And we're hoping that to get some improvements, from ExLibris on this job, and you know I do know that it takes a long time to change workflows and to train people so.
Shannon Pritting: You know I would, I would just kind of.
Shannon Pritting: You know, provide a word of caution to.
Shannon Pritting: Maybe not do that, and leave some months to see if ExLibris can make this job more responsive to the failure connections on oclc and so that there can be a higher level of confidence so things are coming but we didn't want to.
Shannon Pritting: not communicate, but I.
Shannon Pritting: If you're asking for my opinion and guidance, I would not change your cataloging practices to duplicate work in oclc and Alma um you know because.
Shannon Pritting: This should work in the future and what what what I what i've seen is like once you change your workflow it takes years to change it back so this could be you know unneeded redundant work.
Michelle Eichelberger: And I would just say, Brian.
Michelle Eichelberger: I think that's what we're talking about adding holdings so you're setting things up and Alma and you're adding holdings and oclc and you know buffalo is certainly a lot more specialized so maybe you are doing specialized cataloging and oclc but.
Michelle Eichelberger: I think, for the the general person who's doing most of their work and Alma we are encouraging you to not start also adding your holdings and oclc.
Shannon Pritting: Right and and another reason that i'm saying this, too, is, I know that many campuses.
Shannon Pritting: Some of the people who are doing the cataloguing or the you know, the people who are doing the vast majority of the copy cataloging no longer have access to connection, so you know it's a major change, you know, obviously.
Shannon Pritting: Local if you if you know you know what you want to do we're not here to stand in your way, but you know, I think, especially if you're a low volume campus I would not say that there's any need to change that.
Maureen Clements: And Karina has made a comment and there.
Maureen Clements: Might there be a time when we need another oclc reclamation.
Maureen Clements: I think that would.
Maureen Clements: likely be you no, but the future you never know what the future holds so.
Shannon Pritting: yeah from a systemic perspective we don't have any plans for reclamation that would be guided by us, but you know at Albany if you have the staffing to handle the the complexities of of a full reclamation that's that's your local decision but.
Shannon Pritting: we're not seeing that there's any reason to do any groups of reclamation.
Shannon Pritting: Based on our analysis it's just it missed picking up ads and deletes which we know we can restore but a reclamation is is much more complicated.
Maureen Clements: Any other questions.
Maureen Clements: All right, so i'm gonna stop the recording then and thank you guys so much for attending today's session, it was.