ask-annie/out/754690268/transcript.txt

# Transcript: 754690268
# URL: https://vimeo.com/754690268
# Duration: 4816s (80.3 min)

[0:00] Unmute yourself and ask,
[0:03] if we're talking about a topic you are interested in, you have a follow-up question,
[0:08] even if it was not yours originally,
[0:10] or you just want to share how you are doing something in your environment, how you solve the problem we're talking about,
[0:17] how you what you did similar to what we're talking about, just chime in. It's supposed to be a conversation.
[0:24] It's supposed to be about
[0:27] sharing information
[0:29] and details and, yes, answering questions.
[0:33] But
[0:33] I'm personally more interested in getting everyone to engage
[0:38] and get something from that meeting, and that's why I never have PowerPoints or prepared stuff unless someone asks me for something.
[0:46] And with that, who wants to open today?
[0:54] Any other, question if I can?
[0:58] Sorry. Okay. We have two people talking.
[1:00] Okay.
[1:02] So Oh, you got it, Martin.
[1:05] Oh, okay. Thank you. Martin, first, then Danny, I think. Right? Okay. Correct. Yeah. Thanks very much. Look, we have a, unusual request from a client with our ST server. We we push files to the client server.
[1:21] Mhmm.
[1:22] But they want us to delete the files on their server as well. They don't want to do it. So
[1:29] what they want us to do is, every time we log in, we log in once a day, say,
[1:33] and we check the age of the files. And if they're older than seven days, we delete them. That's that's what they want. So they don't wanna do this themselves. It it is a bit unusual. It's very odd, actually. That is there's nothing there's nothing in the configuration
[1:47] that allows you to do this. But is that the kind of thing that's
[1:51] that can be possible through some sort of scripting? I know there's lots of things that
[1:56] ST can do, but I'm not across what it's capable of. Is
[2:01] is that something that could be possible
[2:03] to do with scripting, say?
[2:07] What you so what
[2:08] you so let me think.
[2:11] Technically speaking,
[2:12] what you can do is to implement a custom transfer site
[2:17] and then schedule
[2:18] a pull,
[2:21] not a push. Here
[2:23] were the challenges.
[2:25] When we push a file, once we put the file over there, we can rename it and move it around. But we don't have the capability to delete it because we just put it there. Right? Right. And during a push operation, we don't touch any other file on the file system over there. We don't do a list or anything like that because we literally just we log in, put log out. This is our C plus, nothing else.
[2:51] Right.
[2:52] During a pull,
[2:54] we do a listing.
[2:58] So, if you have a custom transfer site, and I'm just wiggling it here on the fly, so I might hit the door somewhere, I don't know. But if we implement a custom transfer site and you schedule a schedule a pool, for example, every night at three a. M. Or something. Yes. And you go and
[3:17] list the files.
[3:21] Then technically speaking, because you're in custom code and you have an open connection for the pool, you can reuse the connection to delete.
[3:30] It's not orthodox.
[3:32] Right. Right. What about,
[3:36] them complaining that you deleted the wrong files about losing data?
[3:40] Know what I'm saying?
[3:42] Yes. You
[3:44] will not have tracking
[3:45] because we are not pulling.
[3:47] Right.
[3:49] Option is,
[3:50] actually, if you pull those files, so if you implement the transfer site
[3:55] to actually get the files that you need
[3:58] and you pull them, if they're not too big and you don't care about pulling them into your environment, you can actually delete after a pull
[4:06] out Right. Of the I see I see what you mean. Yeah. I hadn't I hadn't thought of that.
[4:11] You know, so if if we're talking about thousands of files, don't do that. You will overkill on your own server for no good reason. Just tell the choosers what they're doing. But if you're talking about five files or something like that, maybe I think it would be. Yes.
[4:26] Maybe worth implementing.
[4:28] And then your biggest challenge will be how to figure out which files to pull so that you can delete them on the way out.
[4:36] Yes. And it it they'd have to be a certain age as well. We would only be we would only pull files that are, say, more than what
[4:43] version of SD are you on? May
[4:47] update.
[4:48] Okay. So look at the later release. So I I think they so what protocol are you using for this customer? Tell me SSH, please.
[4:55] Yes. Oh, yes. Yep. Okay.
[4:58] Look at the new stuff we did for SSH.
[5:01] Because now we have the ability to actually
[5:06] know what files we had pulled based on age and stuff like that. See if you can work out with all that stuff and somehow,
[5:15] I don't know, produce a list and then use the API based on the list or I I I mean
[5:21] Yes.
[5:22] No, that that's a I hadn't Yeah, thought
[5:25] another thing, all of those files actually always pushed by you originally.
[5:30] Yes.
[5:31] Use the API. You have the list. So,
[5:34] basically, because you have the list and you know you push them on Monday,
[5:39] use the API or someone hand waving if there are 30 files, who cares, but produce yourself a list of the files you push to them. Yeah. That's
[5:49] that's that's already helpful. Yeah.
[5:51] And then this way, you have a list for a pull. And you can run with on the schedule in ST.
[5:58] And then on the schedule, you can pull a file from somewhere, then
[6:03] send it into advanced routing. Inside of advanced routing, it has a custom step that does the the the connection to the pool to to to get those files out, you know, something like that. Or you can just use the API from outside. You don't even need the ST for that. Oh, you can. You know, you use ST to schedule the pulls,
[6:20] but the point is you can use another schedule if you have something in your environment already running, like a cron job or m whatever you have. And do it that way. And that way, you know which files to pull, and that solves your problem. Yes. Okay.
[6:35] Great.
[6:36] I'll have a look at it. I'll have look at on the test server to set that up. Yes.
[6:40] But think about numbers. Because if we're talking about thousands of files, remember, the pull is one file at a time.
[6:47] So you'll be pulling all all night long, opening hundreds of connects. Yeah.
[6:52] Yeah.
[6:53] So Great. Yep. If you have that many files,
[6:58] go for a custom transfer site and then implement what so how pool works? We do authentication,
[7:04] then you would do a list.
[7:06] But you when you do custom transfer sites, you implement the list.
[7:10] Yes.
[7:11] During that list, you can do whatever you want, you know? Just use the library to go there and do deletions.
[7:17] Again,
[7:18] tracking will be a problem if you go that way. If you do it with the pause and delete, your tracking will show you which files you deleted. So when they complain
[7:27] Right. You can tell them, guys, that's all we deleted. Whatever else is missing, it's not us.
[7:32] So when when you say custom transfer site, are you just talking about a normal pull or push profile that you create?
[7:40] So so out of the so, yeah, so out of the box, you have SSH and S3 and all that funny stuff, but we also have the plugables,
[7:48] right, where you can install additional transfer sites. And you can write your own. If you pull the developer guide for 5.5, there is actually an example FTP site over there. We'll need to build an SSH one, obviously.
[8:01] Right, right.
[8:02] It's Java code. It's pluggable, which means every it will survive an update. It doesn't need to be rebuilt every time. It's specifically built on top of SPIs. You know, how we have APIs when you're coming from outside. We have SPIs running on the server that allow you to build pluggables. We call them pluggable sites,
[8:21] pluggable transfer sites.
[8:23] If you ask me for example, if you want to do SharePoint, I'll send you to where to download the ready to use SharePoint transfer site, and it is a pluggable transfer site already.
[8:32] Right. The new site's applicable.
[8:35] So you can build your own. That's it's not scripting scripting, it's Java coding, but it's still your option.
[8:43] But, again, I don't know what volumes you're talking about, but if it is a very small number of files,
[8:49] play with the pool. And especially I it will be
[8:53] we haven't actually started this transfer yet, but I think it will be a small number of files they've indicated. So, yeah, discuss with them. If it turns out they're, like, a 100 files per day or less, just do the pull, still decide to not open more than five connections or whatever, run it at 3AM in the morning or whenever your server is not busy.
[9:10] Yeah. You know what I'm saying? Yeah. Yeah. And just,
[9:14] either do it that way or,
[9:17] based on the list or based on the API. Doesn't matter. You know? Play with
[9:23] as I said, I'm winging it at the moment. So I just threw it with you a couple of ideas, see if I need it. No. That's that's I hadn't even thought of that approach, so that's that's great. I'll look more into that.
[9:33] Yeah. Great. Thank thank you very yeah. That's that's fine. Thank you very much. When you're talking about ST,
[9:38] that's actually one of the the good things about it. You can actually reverse some of the questions and send them in a very weird place, and they still work. Right? But, yes. And but,
[9:51] the other thing, make sure they don't those are not very big files. You really don't want to transfer one gigabyte in and out just to delete it. Right. Right.
[9:59] And you'll need to warn them that you they will need to give you rate write a read access if you well, you only need it for a delete anyway. So Yeah. Yeah.
[10:09] But you only need to warn them that if they're on some kind of a plan that charges them for outbounds, that they'll actually see a a full blown download.
[10:18] Right.
[10:19] Right? So talk to your partner.
[10:22] Honestly,
[10:23] if you can convince them to do their own cleanup, that's even better. Well, that's that's what they really should be doing, but, I don't know why they want it done this way.
[10:33] Because I have a lot of different things that they're asking others that normal clients wouldn't ask. So this is just one of these things. Out of curiosity, is it a small company or a very big one that you It's a very big one.
[10:44] It's a very big problem.
[10:46] They need about 7,000 signatures just to get Yeah.
[10:50] Or maybe. In the house. Right?
[10:53] So don't want to customize it specifically for you or whatever or whatever solution everyone else is using is doing the deletions. I Right. I just
[11:03] from security perspective, I really don't like the idea of you being responsible for deleting data on their I know. I know. It just seems totally wrong to me. Yeah.
[11:13] So,
[11:14] I mean, maybe get your security team involved into all that, and get them to talk to their security team.
[11:21] Right. Right. You know, that might actually put key boss on the whole thing.
[11:26] Yeah.
[11:27] Okay. Yep. Yeah. You know, if security can be a pain in the behind when we implement, why don't use them to be pain in the behind when we
[11:39] That's right.
[11:40] Yep. Okay. Well Thank you. Okay. Danny,
[11:44] it's you.
[11:46] Thanks.
[11:48] Now
[11:49] just
[11:51] I don't actually work
[11:53] in the team that manages
[11:55] our secure our ST.
[11:58] Mhmm. It it is it is an X-ray
[12:01] ST that we use,
[12:04] but I'm currently working in the capacity of a business analyst
[12:08] with a team implementing
[12:10] embedded analytics for secure transfer.
[12:13] Okay.
[12:15] Now there is a requirement to get
[12:19] a
[12:20] dump of
[12:22] all the jobs
[12:24] that is currently being handled by our STP
[12:28] instance.
[12:30] But
[12:31] so far, every attempt
[12:34] I tried to to
[12:36] to see if it was possible to get that.
[12:39] So far, it's been impossible
[12:42] for whatever reason. So I'm just trying to figure out, trying to find out
[12:46] if there's a way or if there's currently
[12:50] what are the ways that that are possible where we can get an extract
[12:55] of all the jobs that are currently scheduled
[12:58] and configured in for Secure Transact to manage?
[13:02] That
[13:03] actually is a very easy answer.
[13:06] Not from the embedded analytics side. Not from my side. But on the SD side, there is something called XML export.
[13:14] Yep. So what XML export will give you is an export of all the accounts,
[13:19] and inside of them, all of their subscriptions.
[13:22] A subscription is a job.
[13:24] Every subscription with with a scheduler
[13:27] means that it does
[13:29] a schedule jobs. It it does a pull from somewhere.
[13:33] So it depends on what you need it for.
[13:36] One thing it will not tell you is, for example, if it is now three p. M, what will trigger at four p. M. Because the schedulers are basically cron jobs.
[13:45] So it will tell you how it's configured.
[13:47] So you might need to do some calculation based on that. For example, if it says that it's a scheduler
[13:54] predict every day at 3AM in the morning,
[13:57] It will not tell you that the next one is tomorrow, but you will see that it started on the January 1, and it's running every day at 3AM in the morning. So you'll know there is one at 3AM in the morning.
[14:08] Will it have information such as
[14:11] the description of the job itself?
[14:14] What is it meant to do?
[14:17] Only if someone wrote it.
[14:19] So, basically,
[14:21] and because had you ever seen ST? Probably not. So let me see how to answer that one in a better way. So here is how ST works. Basically, there are two different levels of where you define the job define a job information.
[14:36] One of them is on the pull side,
[14:38] which basically says go and get me a file or a set of files and what kind of files and so on will be in the track in the transfer site itself, which also will be part of this export. So you can track that.
[14:51] So you will see a
[14:54] subscription that says, and I'm going to be pulling from a transfer called Annie, transfer site called Annie. And when you open trans when you scroll
[15:02] to the transfer site called Annie, it will tell that it's connecting to google.com
[15:07] and
[15:08] going into this directory and pulling those specific files, .txt, for example.
[15:14] Yep. Yep. That part is so on the out on the outbound part, on the push part,
[15:21] is question for you. Do you know if they're using advanced routing or basic applications?
[15:27] Uh-huh. Pardon? I didn't hear the first part.
[15:31] Do you know if they are using basic applications or advanced routing? And if you don't, that's okay.
[15:37] Yeah. I'm not sure. I I'll take that note down.
[15:41] Yeah. So if they are using advanced routing,
[15:43] the scenario
[15:45] of what is happening is inside of something called route.
[15:49] And the route contains information with steps that tells you when the file arrives, PGP encrypt it, and then rename it and then send it to this address,
[15:59] to this transfer site again.
[16:01] Right?
[16:04] Now there is a small complication. If they're using it properly with templates, it might be harder to read because there will be variables.
[16:11] For example, the site might be saying, BGP encrypt with the key that is a variable and send to a transfer site that is another variable, and then you'll need to read the subscription to read the variables.
[16:22] But it all the information will still be there anyway.
[16:25] Unless, of course, there is a passing of the file name, and this information comes from the file name, then someone will need to tell you also
[16:33] what the file name means and how to pass it.
[16:36] So this information on the very technical side of the house, you can read from the XML.
[16:42] However,
[16:44] unless
[16:45] someone
[16:46] wrote a note, and there is a couple of places on different objects where you can write notes, including there is the ability to put it as a variable
[16:55] to explain the business logic of why this process is set up,
[16:59] that's pretty much all SD can tell you because we really don't care if
[17:05] this is a finance file or not. You know? We're NFT. We move files around. That's all we do.
[17:10] So Yep. Yep. The technical data, it's there. It can be exported. Someone will need to do some passing of XMLs, connecting dots, and so on, but everything is inside of this single file you can export.
[17:24] Yeah. But it will not give you the business logic. It will not tell you that you are doing that because this is a database backup that needs to move at 3AM in the morning
[17:34] or something like that. That information will unless someone puts it somewhere on the record. And if it is on the record, it will come with export.
[17:42] But unless someone wrote it down or named everything in a way that makes it clear,
[17:48] that information is just not there.
[17:52] Okay.
[17:53] And with the XML export, can that I'm I'm guessing that's a
[17:58] a button that you click and then you get it. Is that something that can be scheduled to
[18:04] to be run? So Yes. Let's say a a regular
[18:08] Yeah. It's a script on the file system. There is a button on the admin UI. Never ever use it to export the whole server
[18:15] because if it times out, it will just get stuck, and you'll need to wait it out, and you cannot stop it in any way or form.
[18:21] But it is Yep. A bin directory
[18:23] of ST itself.
[18:25] There is it's literally called export underscore,
[18:28] x sorry. XML underscore export.
[18:32] Yep.
[18:33] That's literally The the big directory.
[18:36] Yes. That's literally the name. And, you can there are parameters. There is a help page over there. There is a whole page help page on the documentation as well. So you don't need all the certificates so they can exclude them. They can just give you the accounts. If they only export the accounts,
[18:52] it will give you
[18:54] don't use routes,
[18:56] then it will give you everything. If they use routes, you also need to you'll need to export the accounts and the routes that combine the files.
[19:06] Gotcha. There is a lot other a lot other things. There are admin, admin accounts. There are admin roles. There is a lot of stuff. And, again, it's not human readable,
[19:17] really, unless someone parses it and creates a human readable file. But the information is there, which is the important part.
[19:24] Another option is to another
[19:26] option is to go to the API.
[19:29] I personally prefer the XML route for something like that for two reasons.
[19:34] One of them, you get the XML out and you leave the server to do whatever else it's supposed to be doing as opposed to sending API query 7,000 times.
[19:43] And second, XML is
[19:46] structured better for parsing.
[19:49] The API, as good as it is, you know, there is a lot of data, but it's in different places, and you need to follow links all over the place.
[19:56] Yep. With XML, it will basically just it's like one package, and you just go into so
[20:03] almost everything for the single account is inside of an account tag.
[20:07] Right?
[20:08] So you can just read it account by account and pass it on.
[20:12] Yep. Yep. Understood. Yep. I did try the API option,
[20:17] but that didn't yeah. It was unsuccessful.
[20:20] So, yes, I'll try it. I'm not Yeah.
[20:23] You can do it with the API. It just requires
[20:25] a lot more heavy lifting, a lot more parsing, and trying to figure out what is what. No. It doesn't make sense then. And then you end up into a especially if you do routes, you end up into an ID that makes no sense whatsoever unless you know why it is there so you know how to escape out of it.
[20:43] So Yep. XML.
[20:45] So, one thing,
[20:47] you can also export a single account.
[20:51] So if you want, pick up a test account you want to play with and ask them to export you everything for that specific account first and
[20:58] see how the parsing for a single account goes. Once you have that, the other one is just account after account after account after account. You just do them one by one, you know,
[21:08] on the positive side. So but that's about it. And, again, no business information.
[21:15] But Yeah. Yep. Yep.
[21:19] Understood.
[21:20] Thank you. Okay.
[21:21] Good. I'm seeing a raised hand.
[21:24] So
[21:26] where do you put the
[21:28] Sharia? Sharia? How do you pronounce your name? Yeah. Yep. The Sharia. Yep.
[21:33] Thank you.
[21:35] So I have two question. The first one is related to the one you explained with the export
[21:40] on Polar, by the way. Mhmm.
[21:43] I work for New South Wales and we heavily use X-ray.
[21:47] Mhmm.
[21:49] Now the question is,
[21:51] the the challenge I'm I I face day to day is, like, for example,
[21:57] if I have an account and that account have hundreds of
[22:01] transfer sites
[22:03] and subscriptions,
[22:05] okay,
[22:06] subscriptions are going to multiple transfer sites.
[22:10] So it's picks up from one destination and drops in multiple.
[22:14] Now, how can I
[22:16] visualize these things? For example, if I pick up this subscription, I want to see,
[22:21] is these connections
[22:23] with this subscription?
[22:27] Not easily.
[22:29] You you get I've
[22:31] been trying for a lot. I I just give you a heads up. I tried to purse
[22:35] the XML file, but it's become very difficult. Do you have anything, any script
[22:40] handy so that we can customize it and make it easy, or you can help with us?
[22:47] Help us. Okay.
[22:49] Question. Are you using basic application
[22:52] or advanced routing?
[22:54] It depends. It it it is it's,
[22:57] like, it depends on the accounts to accounts. Yeah. And the reason I'm asking is because so, basically, if you if you are using basic application
[23:07] Mostly the basics. Yeah?
[23:09] Yeah. If you look so look at the subscription XML
[23:13] on the basic application subscription.
[23:15] The sites you are pushing
[23:17] to are literally next to each other line by line. They are basically just the references. So this one is the easy one.
[23:24] So if you parse,
[23:27] if you forget about everything else, just look at only the complete subscription tag on the XML.
[23:35] I don't have this handy now, but Well, I know. But, when you look at it, so there are two sections.
[23:41] One of them is the, the in and the out section.
[23:45] Yep.
[23:46] On the out sections, the sites will be basically site reference, the name site reference. Site reference, the name site reference. That's all you need from this XML.
[23:56] This tells you where they are going.
[23:58] The inbound file will be in the in section the same way with the namesake site reference.
[24:03] But if you're using basic application, this parsing is really easy. What you're looking for is inside of a complete subscription,
[24:13] the site references,
[24:14] and into which of the two sections it is. I think I found what I found is is, for example, if I have
[24:21] multiple subscriptions
[24:23] under the account,
[24:25] they come in
[24:27] just
[24:28] I couldn't figure out any order, like, if the subscription
[24:34] has, like, if yeah.
[24:37] Yeah. He's The account has 10 connections. It shows all the connections, but I need to know under the account,
[24:43] I have five subscription.
[24:45] Under the five subscription,
[24:47] I have 10 transfer
[24:49] sites connected.
[24:50] So
[24:51] so one subscription has two transfer sites. One is source
[24:56] and the distension. So I want to see those things
[24:59] again. Yep. Yeah. Okay. Ignore everything else. So when you open the XML, ignore everything else and just pull the the tag called complete subscription.
[25:09] Can you just paste it in the chat, please? Sorry. I'm just, trying to pull up the exported file and see if I can do it quickly on the fly. It's called complete
[25:20] subscription.
[25:21] Like this.
[25:23] This
[25:24] is the tab you want.
[25:26] So if you get five subscriptions, there will be five of them under it, under the account.
[25:32] When you just grab the complete subscription
[25:36] part of XML,
[25:38] and we're talking about basic app.
[25:41] The advanced a little different. But with the basic app,
[25:45] if you open that,
[25:47] there are there inside of it, there will be so look for I I don't remember it. I think it's called site reference.
[25:55] This is a site inside of it that you'll be looking for, but it might be something else like site site something. Something started with site anyway. I don't hear. Okay. Hang on a sec. Let me see.
[26:07] I'm at two different screens or interest.
[26:10] Sorry,
[26:12] guys. If if someone get annoyed with our discussion, please
[26:17] accept my sincere apologies.
[26:21] Let me see what's
[26:24] did you paste in the chat?
[26:26] Yep.
[26:29] So complete subscription
[26:30] and site
[26:32] reference. So
[26:33] so if you open what the your XML with an XML editor so that it can show you the text easier,
[26:40] and what you're looking for is complete subscription.
[26:43] This is the tag. It it is complete subscription as opening tag, and then complete subscription at the bottom as closing tag. So look at just this section.
[26:53] And this is, by the way, how you pass this XML. Basically, you have complete account and then the closing of the complete account at the bottom. And inside of it, you have the,
[27:03] login. You have the transfer sites. You have certificates.
[27:07] You have the routes. You have the subscriptions in separate sections, one after one. So in between this site reference, what I I get, so for example, I got a I just picked up a
[27:19] count.
[27:20] Mhmm. And I when I search with that account, the site reference, it gives me
[27:25] No.
[27:26] Don't look for site reference. First, look for complete subscription.
[27:30] Okay. Give me a sec. Let me pull it out. Okay. As I said, pull only the tag complete subscription.
[27:39] So
[27:42] so after you choose an account or just
[27:45] It it's always inside of an account. Subscriptions are always inside of an account. So you have complete account inside of the account. There is somewhat complete subscriptions.
[27:57] Yep. So I've got the complete subscription tab.
[28:01] Would you like to show okay. Show your screen if if you don't mind.
[28:05] Because,
[28:06] I'm on the Internet, and, my
[28:09] session is on Citrix because I'm working from home. So you cannot.
[28:13] Okay.
[28:14] So
[28:15] I can I can actually hang on? Okay. Do you see so did you find the tab called complete subscription?
[28:21] Actually,
[28:23] yeah. I I think I can copy it. Yeah. I I got the complete subscription folder
[28:27] sorry, the tag under the subscription.
[28:31] It Okay. It says it says many things.
[28:34] Don't
[28:36] you're rushing it. Okay. So can you show screen, or can you copy, or can you if you cannot, either, I can see if I can set up one very quickly. Okay. Hang on a second. I just
[28:51] hold on. My server locked me out.
[28:54] Okay. Let me see if I can.
[29:08] Okay. So I what I
[29:12] for some
[29:15] security purpose, I cannot share these XML files.
[29:20] Yeah. Hold on a second. Give me a second. I'm exporting a quickly, an account quickly just to see if I have something.
[29:28] Uh-huh.
[29:29] And open. Let's see. Do I have subscriptions on this guy?
[29:43] Yes. I do. Hold on. Let me see if I can share something. Share. Share. Share.
[29:51] Yeah. So I can see every complete
[29:54] subscription. Can you okay. Can you can you see my screen?
[29:58] Yes. Just a second. Let me jump into this.
[30:04] Yes. I can.
[30:05] See how I'm showing a complete subscription? This is basically a basic basic
[30:09] subscription.
[30:10] Look at over here where it says transfer configuration.
[30:13] Do you see the partner out and partner in?
[30:17] Yeah. I can see that.
[30:19] Do you see the site reference inside of the transfer configurations?
[30:25] Inside the where?
[30:26] Inside the transfer configuration tag.
[30:29] Yeah. Can you call it in a different color? Sorry. The It is. Do
[30:34] you see it? It says partner out for the tag, and it says site reference to account a.
[30:40] Yeah.
[30:41] So this is your push site.
[30:43] If there is more than one, there will be two site references one after another.
[30:48] So transfer confirmation,
[30:50] configuration
[30:51] is the one which is the push site?
[30:54] No. Transfer configuration,
[30:56] there are two of them. The partner out one is the push sides. Okay. In
[31:02] one, if I don't have a pull here, but if I had one, it will be here.
[31:08] So e one is incoming,
[31:10] so which is coming to us?
[31:12] If they're coming to you, if they if you're pulling from them Yeah.
[31:17] Then you'll be in the partner in. Yeah.
[31:20] And they'll be exactly the same like this one.
[31:23] So in this case, I don't have it. But if I get one, it will be like this.
[31:28] Yeah.
[31:29] Do I get on the fly here? Whatever.
[31:32] And it and then this is the name of the transfer site. So when you parse it, what you how you parse this is,
[31:40] you start with the complete
[31:42] account.
[31:43] Right? It's the top.
[31:45] Mhmm.
[31:46] Right?
[31:47] Then,
[31:48] when the account starts, first is the account tag. You ignore that one because you don't care. Or if you want to grab the name from here, you can. Right?
[31:57] But from the top?
[31:59] Then it's the virtual user tag.
[32:01] Yes. That one, this is basically how the user logs in and so on. Then if there are certificates,
[32:07] they will be here. I don't have any, so I don't have certificate tags. Then you have the sites.
[32:13] Yeah. From here, you parse and you grab the names because then they'll be referenced down. Right?
[32:19] And then start the complete subscriptions one after the other, one after the other, one after the other, each of them looking exactly the same.
[32:27] Basically, this whole tag will repeat and repeat and repeat with the different values.
[32:32] The application reference is what which application you're using, ST. The folder is the subscription folder. Ignore the anchor, that's something for ST, it's internal purposes.
[32:43] And then
[32:44] you are looking, if you're looking for pools where you're pulling data from, look for the partner in transfer configuration.
[32:51] If you're looking for the out,
[32:53] then if it is basic application,
[32:55] it will be the partner out.
[32:59] If it is advanced routing, it will be in the route. It will not be here. So then you will need to find the simple step
[33:07] in the route.
[33:12] Do you have any any any
[33:14] any No. Can you see any simple pass pass? I'm sorry. Nope. Nope.
[33:19] Sorry. No? Whenever I need to pass something, I'll just write whatever I need at the spot. And
[33:25] Oh, okay. Oh, can I,
[33:27] can I share my
[33:29] my one of the example,
[33:33] example,
[33:34] XML file after the meeting? Or
[33:38] If you're asking me if I'm going to write your parser, the answer is no.
[33:42] No. No. No. I'm not asking just to share and see
[33:46] the the thing
[33:47] I want to share with you, not with everyone
[33:51] just to be precisely.
[33:53] Is this possible?
[33:55] Sure. Just send it over. Just keep in mind I'm probably hours behind you. It's, like, 07:40PM
[34:02] over here. Oh, okay. I won't get to it until tomorrow. But if you want to send something, send me a note. We can discuss if anything.
[34:10] Just cannot But, be on the
[34:13] basically,
[34:14] my advice,
[34:15] whatever you are passing this with, find the language or find the library that parses XMLs.
[34:22] And Okay. If you if you look into the I think it's in the call directory,
[34:27] we actually have an XSD file for the for XML for for this export.
[34:33] So if What your preferred,
[34:36] what what do you recommend for pressing?
[34:40] I'm a Java developer originally. So 99%
[34:42] of the cases, I go for Java, but it's too heavy for that.
[34:46] Can parse XML just fine. They test libraries for that. Pro can, although it's old now. So whatever language you are comfortable with, look for an XML parsing library. And you don't remember that you can
[35:01] can't parse, you know, without the schema, but we actually have the schema.
[35:06] So, if you go to the comp folder of ST,
[35:09] it's called, I don't think I remember the name at the moment, XML something called another. Hold on, I might note the name.
[35:15] But the x s this schema, x s g s
[35:20] schema is actually on the server itself.
[35:23] So you can grab it and give it to your library,
[35:26] and then the parsing becomes a lot easier because it knows what to expect.
[35:32] Okay. Okay. Thanks for that. So,
[35:35] Yana, can you share your email address? Or
[35:41] It's How can It should be in the invitation that you got. If you don't have it, pink HoHong
[35:47] will set up the meeting, and he can share it. Okay.
[35:51] Not only secret.
[35:52] I mean, it's just I'll put it in chat. But if you lose it or
[35:57] Oh, that'd be great. Thank you. Much easier. Yeah.
[36:01] I just put it in chat,
[36:03] but if you need anything else,
[36:06] just always think Hong Kong. And the other thing is if you sanitize
[36:11] if you still can't figure out something, you can just just sanitize an example of your file. Mhmm. Here at the community forum.
[36:19] Someone will help you.
[36:21] Yeah. Yeah. Yeah. That's that's why. This is why we are here today. So just to get that Yep. Community help. But,
[36:28] also,
[36:28] you know, writing scripts on the phone is not the best use of anyone's time ever. No. We I'm not I'm not trying to give that, but I'm trying to give my idea and best use cases.
[36:39] Understood. What I'm saying is start your script. If you get stuck somewhere, sanitize an example of your data,
[36:46] post on community on the forum. It's a lot easier to work on written forums on scripts than on the phone. That's all I'm saying.
[36:54] My second question is
[36:55] Mhmm. What we,
[36:57] we are also using 5.5
[37:00] is the and what I found was very annoying that sometimes
[37:06] the
[37:07] subscription
[37:08] fails, like, without any notice, they don't kick off.
[37:12] So the the only solution
[37:14] I can see now is if I delete the subscription and recreate the subscription again,
[37:21] it works.
[37:22] I
[37:23] can't find any clue why it is failing.
[37:26] Okay.
[37:27] What's the usual?
[37:28] Yeah. I have a question. When it fails, do you see in
[37:33] the server lock a note that says that
[37:37] an old run is still in progress or skipping,
[37:41] or nothing happens? It just never triggers.
[37:44] It never triggers. There's nothing I can see.
[37:48] What database are you using?
[37:50] What updates?
[37:52] What database?
[37:54] Oracle?
[37:55] MySQL.
[37:56] Oh, Oh. What update are you on?
[38:01] Sorry. What's the question again? Sorry. What update? What version of test? You said five five, but which update?
[38:08] I think the the basic one we have, we are in the process of upgrading that.
[38:13] So I'm just in the process of, put a change request and update it. So the base base base,
[38:20] we are using now 5.5.
[38:23] Ugh. Okay. So that's a two years old
[38:26] build Mhmm. At this point.
[38:28] And, yes, I do remember that, and that's why I'm asking this question occasionally, because some of the early May did have this problem. They were basically So losing their own
[38:39] update to the newest version or close to the newest.
[38:44] And if you still have the issue, open a ticket with support and give them as much details as you can,
[38:51] because
[38:52] there are two possibilities happening here.
[38:55] One of them is that the cron job is not in the database at all for one reason or another.
[39:01] The other one is that it's misconfigured, so it doesn't fire up.
[39:07] If
[39:08] it happens again,
[39:10] after the update,
[39:12] fix it.
[39:15] Right? Don't touch this subscription. Create a new one in a new folder temporary. Do something else to get your files going.
[39:22] But don't touch the existing subscription and get support to look into it. It's possible they might spot what's going on. The moment when you delete the recreate,
[39:31] you know, I we'll need a crystal ball to try to figure out what happened in your environment.
[39:37] Mhmm. So,
[39:39] but it shouldn't happen, obviously.
[39:42] But, usually, when it doesn't happen and you don't see the line in the logs saying,
[39:48] another transfer is an earlier
[39:50] one is the learning,
[39:52] it usually means that something got corrupted in the database.
[39:57] Yep. That's a good, good tip, so I'll keep that in. And
[40:02] the reason why when you recreate it works is because when you so this unlike all the other objects in the database or almost all of them, subscription gets new IDs when you create them.
[40:13] So,
[40:15] when you delete the old subscription and create a new one, the new subscription has a new ID, which means the scheduler attached to it has a new ID, the cron job has a new ID.
[40:26] That's why when you try to fix a subscription that exists already and you keep working on it, it might not fix it because if it is corrupted, it doesn't save properly anymore,
[40:37] and it's the same ID. When you recreate, even though everything else is the same, the ID is different, so it's a brand new object for us.
[40:45] Yeah. I, I noticed that even if I put the,
[40:50] what's called, same,
[40:53] name, same subscription subscription name,
[40:55] it works. But it has to be deleted and recreated.
[40:59] Yes. And the reason for that is because when you create it again, it gets a new ID.
[41:05] From there, the name of the schedule, the way we name the and the schedules is based on the ID, not the name of of anything.
[41:12] So that's what's going on.
[41:14] Oh, is there any any other
[41:16] other
[41:17] quick
[41:19] quick,
[41:19] commands that I can refresh the ID or something? Because
[41:25] And that's why, as I said, after you update, if it's still happening, get support to look into that and try to figure out why it happens.
[41:33] Because without the root cause,
[41:35] literally, there is nothing anyone can help with.
[41:40] Sometimes it's a corruption. Sometimes it may be an API call going wrong. Sometimes it may be
[41:46] a book. You know? Things happen.
[41:49] And sometimes you're just the unlucky guy that just happened to know the most important file. It happens.
[41:55] It is happening. Yeah. Like, some of the files are pretty critical, and we only notice when the user complains that they didn't receive the file.
[42:04] And that gets p one ticket. I
[42:07] I understand.
[42:08] And
[42:09] believe me, whenever a root cause is found, it got patched. And I know that they had found a few times when schedules are going crazy with earlier versions of 5.5.
[42:19] So
[42:21] keep an eye on it. If it happens again, especially after you upgrade, get before you recreate,
[42:28] before it does the subscription, get support to look into it.
[42:32] Okay.
[42:33] Thanks for the tips. Yep.
[42:35] Yep. I wouldn't take much time. Thank you. I have yep. Good. I have and I cannot pronounce the name. Is it Chiani Chiani Chiani? How do you pronounce it? Hi, Annie.
[42:45] Chiani.
[42:47] Chiani. Okay. Yeah. That's cheap. Okay. Okay.
[42:51] Sorry. I have a lot of questions.
[42:53] I will paste in the chat.
[42:55] Go ahead.
[42:56] So so my first question is, if I configure all the transfer flows under one single ST user account,
[43:03] will there be any problem?
[43:06] Yes
[43:07] and no.
[43:09] Because
[43:10] Technically
[43:12] technically,
[43:13] there is no limit,
[43:15] but the loading of the pages on the admin UI can go really, really slow.
[43:22] And the login when when this user needs to log in in order to put a file or get a file, this login and list and everything will be slow because there will be way too much data to go through.
[43:35] Now Okay. If you go that way, we actually did a couple of so what version of ST are you on?
[43:44] I'm sorry. Again? Sorry. What version are you on?
[43:50] ST 5.5
[43:51] right now.
[43:52] Which update?
[43:54] I think just upgraded to
[43:58] May, maybe.
[44:00] Okay.
[44:01] Yeah. You may want to look at the release notes of the latest releases. I don't remember when we added those, but we added two special updates
[44:11] in the last year or so
[44:13] specifically for accounts that have a lot of subscriptions.
[44:18] One of them is how to look
[44:21] up the,
[44:24] folder order.
[44:26] Do you want to go recursively or linearly? It depends on if your folders are next to each other or if they're
[44:35] under each other.
[44:38] Okay. So basically, for this kind of account, with you said about what? About 130
[44:43] partners, so this will mean, what, 130 subscriptions or more, you may want to convert into a crisis. The out of the box way is iterative.
[44:53] So it will be, if you open the subscription
[44:56] itself, there is a checkbox.
[44:58] I don't remember if it is there in May yet.
[45:02] Oh, so there's a checkbox to,
[45:04] even choose how we want to search for the subscription, is it? Yes. Because otherwise,
[45:10] if you read it on how it is working out of the box, every time a file is uploaded,
[45:14] we will read every single subscription,
[45:17] folder to see where you might be even though we know where you are. It's a little stupid.
[45:22] But that's how it's implemented.
[45:24] And for it doesn't it doesn't make much difference if you have three subscriptions,
[45:28] but on 130
[45:30] it will.
[45:31] The other thing is every time when the user logs in, one of the things that the ST is supposed to do is to check if all the subscription folders are created or not,
[45:42] and create them if created.
[45:45] As you can imagine, with 130,
[45:47] that's a lot of things. So there is a new parameter, and this might be after May,
[45:52] that actually will tell the server
[45:55] that the subscriptions
[45:57] are pre created or something along this lines. I don't have the name in front of me.
[46:02] But open the release notes of the latest release, and when you read it, you'll see it. What it does is basically to tell the server that if
[46:12] it had checked in the last interval,
[46:15] which is usually in the last few minutes, don't check again, but assume that everything is still in place.
[46:22] Okay.
[46:23] So even if
[46:25] even if we
[46:26] pull for every five minutes, that kind of situation is also supported.
[46:32] Yes. It's okay. It doesn't from ST perspective, ST really doesn't care if all of the pools are on the same account or not.
[46:40] Oh, okay. Because
[46:41] for us, a pool is a pool. We really don't care if they're on the same account.
[46:46] But usually
[46:47] the problem comes from the file system and
[46:50] just the amount of folders and subfolders and data in the same place.
[46:55] See, and that's why that's where you need to be careful.
[47:00] And
[47:01] the pools themselves will be okay. I'm a little worried about the file system, although you'll be in the separate because
[47:07] you'll be in subfolders under the same main folder. And when this user logs in,
[47:12] they'll be getting into they'll need to list this whole thing, the whole tree. Right?
[47:17] That's where my worry is. That's where your slowness might start coming from. It will not be from them being.
[47:24] So
[47:25] one option might be to actually create more than one account depending on which point it is or where it's going. Remember that with the push you can use a site from another account now.
[47:36] So you can do a couple of special accounts,
[47:40] or a few special accounts, each carrying 20 or 30 posts or whatever, and then just distribute where they need to go or something along those lines.
[47:50] Just as but I
[47:52] mean,
[47:53] the reason why we did these two, changes that I was just talking about, I have a customer that is running 13
[48:01] they at the moment, we are down to 400 subscriptions,
[48:04] but they have 13,000
[48:07] folders
[48:08] I see. Under
[48:10] the whole folder of the user.
[48:13] And you can imagine
[48:15] just, you know, every time we go to these 300 subscriptions and sometimes to the 13,000
[48:20] folders.
[48:21] So that's the scenario that worries me.
[48:24] Just keep an eye on that so it just doesn't grow too much. Other from that, from pure ST perspective,
[48:29] we really don't care
[48:31] if you're on one account or 25.
[48:34] Okay. So our customer having issue because
[48:37] they are we are still checking whether it's a shared storage problem or I'm suspecting is it because of one account having multiple
[48:45] subscriptions.
[48:47] What kind of issue?
[48:51] Like, intermittent issue
[48:54] to
[48:55] retrieve file or
[48:57] some file start with add process at
[49:02] append to the
[49:04] file name.
[49:06] Uh-huh.
[49:07] And some yeah.
[49:09] Yeah. This will not have anything to do with them being on a single account unless
[49:14] and if you have a 100 and something, that's not even a big account in in the big things things.
[49:20] So then should be some some other issue.
[49:23] Yeah. It's possible that your cash sizes might need to just think a little bit for the account if you have too much information in it, but it doesn't sound that it's that big.
[49:34] Okay.
[49:35] Understood.
[49:36] So
[49:39] my second question is,
[49:41] during migration,
[49:43] is there any configuration not able to export by the XML export? For example, like,
[49:50] I remember for the restrictions
[49:53] in the web UI, I'm not able to export using the XML underscore export. Yeah.
[49:59] There is another so what are you migrating to? Is it a different server or a different
[50:05] Different server.
[50:06] Because they want to change the the OS.
[50:10] Ah. Yeah. So, and here is where
[50:14] so, technically, there is another export, which is the system export.
[50:18] But the problem is that the system export is tied to the OS a little bit. So if you're changing OSs,
[50:24] it it might backfire on you.
[50:28] For
[50:29] restrictions and so on, actually, the easiest way will be through the API.
[50:36] Oh, there's API to do the? Yep. From the API, you can read them
[50:42] and, you know, use a get and then reformat a little bit to do a put.
[50:48] Okay.
[50:48] On the other side. Because are five five already,
[50:52] So the whole access menu, you can read with,
[50:55] with API and actually
[50:57] put on the other side with the API.
[51:00] That will that will be the easiest way.
[51:03] I see. And another thing is I saw there is a export in the server configuration page.
[51:08] But what I that's the one. Yeah. That's the one that I call system export.
[51:13] Okay. But we should not simply import into the new setup. Right?
[51:18] No. Because when you export, it exports files,
[51:21] and it exports other data
[51:23] that is put there during the installation, and it depends on the on the OS.
[51:29] Mhmm. So,
[51:31] technically speaking,
[51:33] it's supposed to be used for recovering your own server if you need to rebuild or stuff like that.
[51:40] This is what it can be used for. When you go to another server, it's possible that you import a configuration that basically will blow the whole server away because
[51:49] it's I don't so what what when you say OS change, is it just upgrading to OS, or are you changing the OS? And what you're changing from and to?
[51:59] Change the server. I mean, it's a new server, completely different server with a new Yeah. With a maybe 2,019
[52:06] OS, something like that. Yeah. Yeah. Especially because it's on Windows, don't even try to use system export around.
[52:14] Okay. By
[52:15] the time
[52:16] you get yourself out of the hole, you put yourself in with the system export,
[52:21] you'd have done it by hand,
[52:24] honestly.
[52:25] I see. But look at the API. The API actually can you can get the list with get, and then someone can reformat for a put, or, you can do some scripting. It depends.
[52:36] I see. Okay.
[52:38] K.
[52:39] And sorry. My third question is
[52:42] how to advise customer which shared storage to use? Because when I see the documentation, it's, like, very simple, like, very hard to yeah.
[52:52] You're on
[52:53] Windows. Right?
[52:55] Either Windows or Linux. We have both customers
[52:58] using both OS.
[53:01] It depends on volumes. It depends on how many servers they have. It will depend on a lot of things. Basically, for we for Linux,
[53:10] the best option is GPFS, the IBM system.
[53:14] It's also the most expensive,
[53:16] so almost no one has it.
[53:17] GF if they're on Linux of some type, GFS two is an option.
[53:24] NFS, good old NFS,
[53:26] even though
[53:28] it might be
[53:30] less performant
[53:31] than the clustered options,
[53:34] it actually
[53:35] occasionally
[53:36] more often than not is more than enough for everyone, and it's easier easier to administer.
[53:42] So when you need to determine what to use, there are two things to do on Linux. Windows is a separate daily week altogether.
[53:50] One of them is look at the capacity guide.
[53:53] The capacity guide contains tables that compare performance of different operations
[53:58] on different file systems.
[54:01] Depending on what is your basic use case and where your major
[54:05] majority of your use case is at, you do some calculations to see if you can fit under
[54:11] the performance metrics for the file system that you chose.
[54:16] For most environments,
[54:17] and I'm not joking, probably 90% of the environments are out there run on NFS.
[54:23] Even for Windows is using NFS?
[54:26] No. On and that's for as I said, that's for Linux. For Windows,
[54:30] most of them are just running the good old slash slash name of the server and so on. You know? The thing that translates as SIFs on if you move it to Linux,
[54:39] it's basically normal Windows storage,
[54:42] and and it there, NTFS,
[54:44] whatever.
[54:45] My brain is So it's NTFSR.
[54:48] Windows.
[54:49] Yeah.
[54:50] You know you know where you don't mount anything, but you have the slash slash, the name of the server, and then folders and so on, and Mhmm. You know, access to the vault. That's usually the fastest way on Windows simply because Windows is not built to work that way.
[55:05] I see. Is there, like, a very simple table to see which category we are falling to? Like, is it small or medium? Or
[55:15] How many transfers do you have per day?
[55:19] Oh, this is the difficult question that customer can't answer.
[55:23] Well, and that's that's where the answer comes from because where
[55:27] you fall in will depend on the number of transfers. If you're talking about thousands of transfers per day, you know, NFS will support you just fine.
[55:37] You are talking about is okay.
[55:40] If you are talk we're talking to Linux. Right?
[55:43] If you are if you are talking about millions of transfers per day,
[55:48] then you might look at something else. On the other hand,
[55:53] two of the biggest customers in The United States are actually using
[55:57] NFS.
[55:58] I see.
[56:00] So it also depends on what you have. There is no point of you recommending a file system if the customer doesn't understand it and cannot administer it.
[56:10] Okay. So it's based on volume of file transfers?
[56:13] It's based on volume. It's based on
[56:17] volume of traffic as well, not just how many files, how big are the files.
[56:22] It depends on what they're doing as well. Some of the file systems are better for one thing than others.
[56:29] And don't and
[56:30] the
[56:32] default position is NFS.
[56:34] However, there are better ones.
[56:36] GPFS and GFS two are always going to be better. If you're in a cloud situation,
[56:41] there is a lot bigger list in Amazon
[56:44] and so on.
[56:45] So,
[56:47] unfortunately,
[56:48] I cannot tell you this is the best one everyone can use it, and it will work for everyone. This just cannot,
[56:54] you know, that's not how it works.
[56:58] So the best way to do it, and what I usually do when people ask me that question, is to literally get their volumes,
[57:05] get their scenarios.
[57:06] Are they just going to upload
[57:08] the file and push it somewhere else?
[57:10] Or are they going to the PGP and so on, and then open the capacity guide and see
[57:16] based on the number of servers and based on whatever actions,
[57:21] how many
[57:22] transfers can each of the file system support in an hour, for example.
[57:27] And see what matches. If it turns out you're overgrowing the NFS, then obviously it's not NFS. Also, if you are talking about a cluster with more than four nodes,
[57:36] NFS is nope,
[57:38] never ever,
[57:40] because it will kill you in terms of how it works.
[57:44] Should be less than four nodes for the NFS?
[57:48] Technically less than three. By the fourth node, you are starting to see,
[57:52] diminishing returns,
[57:54] although it depends on, again, how much traffic do you have. If you have the fourth node just to make sure you always have three processing, then NFS might still be fine.
[58:04] But,
[58:05] again,
[58:06] open the capacity guide. There is very nice tables that literally show you data,
[58:11] results from this file system, this file system, this file system next to each other for each operation.
[58:19] How it behaves on upload, how it behaves on download. And it's not just file systems, it also shows you databases because it's also important. So you can see
[58:28] how it will behave with Oracle and NFS, compare it to MS SQL and GPFS two, and, you know, whatever combinations you can think of.
[58:38] Again, not straightforward. You you cannot just plug the number somewhere, and it will magically create a numb a name for you just because it's not linear.
[58:49] It really depends.
[58:52] K. Understood.
[58:54] Okay.
[58:56] And I see you have more questions. Yeah. Last two questions is, like, what is the best practice when doing fine tuning and capacity planning?
[59:04] Because sometimes it's quite
[59:06] challenging
[59:07] for us to just look at the guide, but not sure how to start with.
[59:13] So there is an article
[59:15] on
[59:16] the on the support portal called fine tuning for ST or tuning for a secure transport,
[59:22] which is a very good start.
[59:25] It basically
[59:27] contains parts of what is in the capacity guide, but explained a little bit less
[59:33] without all the graphics and everything. It's just a list of things to apply.
[59:37] What you need to be careful about, though, is not over tuning the server
[59:42] because sometimes more is less.
[59:45] If you give it too much memory, for example, and it needs to index all that memory and your peaks are coming only once a day, Chances are that at the time when the peak comes, there won't be memory available, and the garbage collector will need to start working at this point, which is the worst case scenario.
[60:01] So
[60:04] sometimes it's
[60:05] high
[60:06] and
[60:08] see how it behaves and change it here and there. It's an art. It's not a science. But the way I start usually
[60:15] is
[60:16] the tuning article in the support site as a base,
[60:20] and then listening to the customer on what exactly they are doing, and from there going to the relevant tables
[60:27] on the capacity guide. So if they tell me I'm an Oracle customer,
[60:32] you know, enterprise cluster with Oracle or NFS,
[60:36] and I have a lot of PGP,
[60:39] I will go to see
[60:40] in the capacity guide what are the metrics for that and what configurations have been applied for that.
[60:47] Mhmm. It's
[60:50] it's a lot and in most cases,
[60:52] you you will never succeed from the first time to fine tune the serum.
[60:56] You
[60:57] the goal at the very beginning should you to give you a good baseline.
[61:01] So using the basics and using
[61:04] using, you know, the basic might be saying three gigabyte, but you know that they're using way too much SSH, so you might want to bump that up.
[61:11] The basics can tell you put this two gigabytes on FTP, but your customer never uses FTP, so why don't you bother, right?
[61:18] So,
[61:19] based on that as a baseline, then take your baselines and figure out what is slow what is slower.
[61:26] But for that also, the customer will need to have a test shield in a way. You know, if they are going to do a lot of uploads and pushes, you want to have a couple of accounts that do that, a lot of that. Right?
[61:39] If they are doing a lot of pitch key, same thing.
[61:43] So is it after the fine tune we need to do some load test to check whether our fine tune Oh, yeah. Values is correct?
[61:50] Yes. And it will double the first step 10 times.
[61:54] I mean, if I've I've probably tuned more servers than anyone else on the phone, and I cannot get it on the first time either. No one can.
[62:03] Because it depends on how your file system responds. It depends on how your database responds. It depends on where the partner is and what kind of timeouts start to play. It depends on the scenario.
[62:15] You know, You fine tune a little bit, and part of it is to also understand
[62:19] that
[62:20] you need to fine tune for everything, not for a specific scenario. Because what I'm seeing a lot when people start fine tuning is they concentrate on a specific two or three scenarios,
[62:31] and they fine tune the server so those three scenarios is as fast as possible.
[62:35] What they forget is that there will be at least five more scenarios on the same server,
[62:40] right?
[62:41] And that fine tuning might actually slow those a lot more. So sometimes the best case scenario for a specific
[62:49] use case
[62:50] is not how you want your server to be fine tuned.
[62:54] Mhmm.
[62:55] Right?
[62:56] Plus don't forget that you also have control on fine tuning on the subscription level and transfer site levels when you're doing pulls and pushes. You can determine how many connections to open, how many fast moving in
[63:09] right? All that stuff is also part of the tuning. Once you tune the server itself,
[63:15] then you still need to tune the jobs themselves.
[63:19] For example,
[63:21] don't put all your schedulers running at 3AM in the morning.
[63:25] Right? Don't compete to each other. Try to move the schedulers away from each other so that they run
[63:32] throughout the whole night, day, whatever it is.
[63:36] If you're sending thousands of files and drowning someone else's servers because of the way we're opening one connection profile,
[63:43] work with your partner and make it a smaller number. Now, for this specific partner, the moving of the file will be slower,
[63:51] but you'll have less errors,
[63:54] which will mean that the server can actually do more jobs because it doesn't eat threats for these error threats.
[64:01] See where I'm going with that?
[64:04] And
[64:05] slowing down something
[64:07] will help everyone else getting faster, and that is worth it.
[64:12] Understood.
[64:14] It's a balance, and that's why you can never get it. And the most important part, if you fine tune it and it's beautiful and truly keep an eye on it, I can guarantee you will need to do something in two months to adjust
[64:27] Because your system will keep changing, people keep changing,
[64:31] transfers keep changing, you keep adding more and more and more, right?
[64:35] So when your bounces start shifting, you just keep shifting, and sooner or later. And, and again, don't forget to play with the jobs themselves.
[64:43] Think about folder monitors. That's my favorite example. You know, the out of the box, if you just run a folder monitor, it will run every five seconds. Well, it runs all the time, but it checks for a file every five seconds.
[64:54] If you have a folder monitor that you know doesn't deposit more than one file per hour, make it a schedule for the folder monitor.
[65:02] Run it every hour instead of every five seconds.
[65:05] Right?
[65:06] On the subscription level.
[65:08] If
[65:10] you know that
[65:13] you are fluid, again, the error condition on the partner side and so on, keep an eye on your and the server will try to help you as well. If you still start seeing a lot of errors in specific cases, see if you cannot fine tune the scenario, not the server,
[65:29] like reduce the number of connections
[65:31] or
[65:32] something along these lines.
[65:37] Makes sense? I know that that's not the answer you want, and I know that the best case, you know, the best answer here will be, and here is the list. Do yes, and everything will work perfectly.
[65:48] Yeah. If it was that easy. Yeah.
[65:51] It's not easy.
[65:52] No.
[65:53] And it's
[65:54] I mean, that that's what I'm doing day in, day out, basically. I'm fine tuning
[65:59] complicated customers,
[66:01] and I usually get called when things go really, really wrong.
[66:05] That is basically part of what I'm doing here, actually.
[66:09] And,
[66:10] usually,
[66:11] it's
[66:12] in my experience, whenever things are really going wrong, it's because someone of
[66:17] undertune
[66:18] but overtuned.
[66:20] They make him to work for a specific set of scenarios and forgot that. Yeah, 90% of the files are not in that scenario, actually.
[66:30] So
[66:32] that's the best you can do. And keep an eye on our server work. That's the other advice I can give everyone that is working on servers.
[66:39] Every day check your errors, see what the server is complaining about.
[66:43] If you have a problem, it will start showing up.
[66:46] That's your first.
[66:48] You know?
[66:49] Sometimes a single transfer failing tonight, failing another one tomorrow, maybe you need to go and check what's going on.
[66:57] Even if there were retries and they went through and the file actually got delivered, if you start having errors,
[67:02] maybe you need to fine tune this scenario itself.
[67:07] Make sense? A
[67:09] little bit Yeah. At least.
[67:11] Yes. Thank you so much for the explanation.
[67:14] Yep. Okay.
[67:16] What else do we have?
[67:23] Anyone else?
[67:30] Everyone is very silent on this phone.
[67:37] Oh, we have about fifteen minutes left.
[67:40] So,
[67:42] last chance, if no one wants to talk, then
[67:46] I can go have dinner.
[67:51] So
[67:54] anyone else? Additional questions, new questions, someone that can Sorry,
[67:58] Annie. I'll I'll ask one.
[68:00] Since we we've upgraded all our MFT sites to 5.5
[68:04] in the last six to twelve months,
[68:08] and one thing I've noticed is that
[68:10] when we do reboots of these servers, these are all Linux servers,
[68:14] sometimes all the services don't start up.
[68:20] And we've and and Yeah. We start them. Okay.
[68:24] Are you on Windows or Linux? Linux.
[68:27] Good. And this is the May they're all on the May update.
[68:31] Are you enterprise cluster or standard cluster?
[68:34] Standard.
[68:36] Okay. And, also, they're all Linux eight servers.
[68:39] Okay.
[68:40] And in your start up script, are you calling start all, or are you calling
[68:46] the different services one after the other? The the different one. It it depends on which sites. Some have FTP,
[68:52] some have HTTP.
[68:54] They're the they're the two things that vary. Everything else is usually started.
[68:59] Yeah. That's because,
[69:01] I know what the problem is. That's a known issue, and you usually solve it by putting a
[69:07] sleep at the beginning of the script.
[69:09] So here is what's going on. The database reports a target,
[69:14] but it's not yet.
[69:16] So an HTTP, which is a very quick service to start because it's basically just an Apache handler. Right? Right. Tries to get to the database.
[69:25] The database is not ready for it.
[69:29] So the trick I used to do
[69:32] is,
[69:34] and that's why I asked you if you're calling all the services one after the other, or if you're calling the start all script.
[69:39] The start all script actually accounts for who is slower and faster by the order of the different processes, but depending on what is disabled, that might not take.
[69:48] The easiest way to solve,
[69:51] at the end of the start DB script,
[69:53] put the sleep for about
[69:56] a second and a half or two seconds. That usually is enough to clear the the problem.
[70:01] I'm sorry. Where do you put this?
[70:04] Open the start DB script
[70:07] in the bin folder.
[70:09] Either in the start DB script itself, or you put it or you can put it on whenever your service are are starting in the Linux because now you, you know, you're making it manually anyway.
[70:19] So just on the startup,
[70:22] when during
[70:23] the
[70:25] problem is that the HTTP starts too fast. That's your problem. You want to slow it down.
[70:29] Right.
[70:30] So you either put it at the beginning of the script just before the start HTTP,
[70:35] or you put it,
[70:37] after the database startup or something like that. But you just want to slow it down a little bit. That's your problem. And because those servers are faster, it they get the process faster. And the five five is faster, so it starts faster.
[70:50] You know, one of those situation where
[70:53] unknowingly enough, being faster is not good. So put to sleep somewhere.
[70:59] So this this is so I'm just trying to think of where this goes. This is
[71:04] are you talking about where where to start all script, for example? The start list or all? Can do your start list.
[71:13] This is so you can put it on top of the start list, or you can just open
[71:18] the start DB script itself.
[71:21] Right.
[71:22] Or the start HTTP script itself. You know, in the same place where the start off script is, there are separate start scripts.
[71:28] So when you call start o, it just calls the other scripts one after the other. Yes. That's all it does. So you can just open the start HTTP script, and at the very top of it, just put a sleep.
[71:40] Or you can do that inside of the start all, or you can do that only on the
[71:46] whatever you are calling the start all for the automatic start up.
[71:51] Right. So sorry. You said put what was that you said? Put what?
[71:55] You need to sleep. You need to slow it down.
[71:58] Right.
[71:59] That's all.
[72:01] Get your OS admins to help you if you are not scripting first. You mean, like, a delay? You put in a delay at the start of the script? Is that what you mean? Yeah.
[72:10] Right. Okay.
[72:11] Basically,
[72:12] what happens is HTTP starts,
[72:14] either the database or the networking of the system is not ready for it yet, so it gets stuck. And because it's an HTTP service, which is stateless,
[72:23] it doesn't know it is stuck.
[72:25] Right.
[72:26] That's why we don't have this problem with SSH. SSH is an intelligent demo.
[72:30] The protocol itself is intelligent enough. It knows what it's doing.
[72:34] Stateless protocols like HTTP are a little problematic this way.
[72:40] Right. Right.
[72:41] Another option, and that's what I and part of the problem is that when you when it starts wrongly like that, sometimes you just need to kill everything and then go into kill minus nine on a couple of processes because you cannot stop them because they're stuck.
[72:56] So that's why I like this solution.
[73:00] Right. Okay. Okay.
[73:02] It's Yeah. Because because usually every six or eight weeks, we do an OS upgrade. We patch our next server. Studies are rebooted as
[73:10] part of the update, and this might be, like, 02:00 in the morning on a Sunday morning, whatever.
[73:14] Yeah. And people people do get called out to fix them. So yep.
[73:20] And and that's why, as I said, put sleep over there. It will slow down the startup with a few seconds, but who cares? No one will get the alert because it will start anyway.
[73:30] I thought I'm I'm sorry. Put this delay on the ones that are not starting up,
[73:35] like HTTP?
[73:36] Okay. Because we have had it with SSH as well.
[73:39] Well, if the SSH is not starting, that usually means that you have networking services not starting fast enough on the file system. So you might want to also delay the whole start all.
[73:50] You're right. Yeah. So, I mean Okay. I'll I'll look at that. Yeah. Based on your In this in this particular case, we don't have any we we don't have any NFS, but it it is there is networking involved, I guess. No. No. It's not the NFS. When you restart
[74:04] when you reboot the server,
[74:06] you know, the server the OS needs to start the networking services and all that funny stuff behind the scenes. Right? Right. Right. And it starts too fast.
[74:15] Right. So by the time we reach for an interface, the interface is not ready from the OS yet.
[74:21] Right.
[74:22] That that's the secondary problem. The you know, either the database our own database is not ready, or something called the OS level is not ready. Right. Both of them are solved by just wait a little bit. Okay. Okay.
[74:35] You know, it Well, I've got a I've I've got a clone taken of the server that failed at the weekend, so I might just play around with that, put in some delays and so on, like you said.
[74:44] Yep. And, you know, it will just slow a little it will it will just start a little slower, but at least you will know that. Oh, that's okay. It doesn't matter if it if it takes ten minutes to start up because it's it's it's only something that we do every every six weeks or so. So having an extra time. Yeah. I'm not talking ten minutes here. I'm really talking about seconds.
[75:02] Right. Oh, yeah. Yeah. No. Not yeah. You know, we're just just
[75:06] give the OS a little bit more time because we're just way too fast out of the door.
[75:12] Yep. Yep.
[75:14] Okay. You know, that's why on Windows, for example, that's why I asked you what the file system what OS you are on. Windows,
[75:20] we advise people never ever to have the services,
[75:24] the SD services,
[75:26] on automatic
[75:27] startup because they start in the wrong order.
[75:30] Instead, they need to create a new service with a start all inside of it and a sleeper before that.
[75:37] Right. So that starts, it waits for the OS to connect everything properly,
[75:42] and then do a start all, which will get the correct order out of Yeah. Yeah. Which on Linux is already happening because you don't have services, you know, that's why you're calling start all automatically.
[75:53] But you still need the sleeper. So Right.
[75:57] It's an old thing.
[75:58] And I Okay. Don't
[76:00] add it into officially into the product because it doesn't it's not always needed. And if you add it there, people ask you why. The answer is because we are too fast, and your OS is slow.
[76:11] Right. Right. And it depends on what they get loaded on the OS. You know, some OSs can be booting for five minutes, some might be in two minutes slot. Right? Right. The slower the OS boot is, the more likely is that,
[76:25] in the sequence, we'll start way too early for the OS to be ready for us.
[76:29] That's my experience. So
[76:33] Okay. Great. Thank you.
[76:36] Okay. Anyone else? Anything else?
[76:41] I have another question. So how can we do the load test?
[76:45] If is there any tool that can help the load test?
[76:51] There is a lot of tools out there. Most of them cost a lot of money.
[76:57] Any free tool that we can use?
[77:00] Or how you do the look test normally?
[77:02] Yeah. So internally,
[77:04] next way, obviously, we're paying for some of the expensive tools. There is a document somewhere describing what we are using.
[77:11] If we're talking about just testing a quick test with a single scenario,
[77:16] just write a script.
[77:18] I mean, send 10,000 files into SG. Just make sure you have enough space for them.
[77:23] You know? Open Oh, okay. Open 20
[77:26] connections and feed
[77:28] 10,000 files in each at the same time.
[77:31] You know? That kind of stuff is not that hard to write with just batch scripting or Windows scripting if you are on Windows.
[77:39] Okay. I got it. Thank you.
[77:41] Yeah. Now if you want to do a real test with heavy,
[77:45] you know, and metrics and so on, that's more complicated. But if you're just trying to see if the server will handle the 10,000 files in 10 connections or whatever, just stupid.
[77:56] And,
[77:58] and don't get me wrong. There probably are open source
[78:04] packages over there for that. I don't know.
[78:07] Okay.
[78:09] Whenever I do any load testing, it's either the internal staff with our tools or the customer providing the tools.
[78:17] So I see.
[78:20] And some of them can be expensive.
[78:23] Some of them can be on the cheap side. So,
[78:26] look around. Just blood test. You
[78:30] know what? It's actually a perfect question for the community forum.
[78:34] Ask people how they test.
[78:37] Okay.
[78:39] Yeah.
[78:39] People will show up and give you some ideas. Maybe someone has a free package that they are using that I haven't heard of
[78:46] or have used. Yeah.
[78:48] I'll check with the community.
[78:51] Yep.
[78:51] K. Thanks. Thanks, Annie. Uh-huh.
[78:54] Okay.
[78:56] What else do we have?
[79:05] None?
[79:06] Okay.
[79:07] Still around?
[79:11] Yep.
[79:13] I think we're done here. See? Okay. Left so we can do your closing speech.
[79:20] Yes. Thank you, Amy.
[79:23] So,
[79:25] if no question, we will see all of you again hopefully, right, next quarter.
[79:30] Right? And then after the call, we'll send you a survey. So
[79:34] if you can, please help us answer
[79:37] give us your feedback.
[79:39] And then there will also be
[79:41] recording posted on our community portal.
[79:45] K? If not, thank you all for your time today.
[79:50] We will speak again in
[79:52] the next round.
[79:54] Thank you.
[79:56] Thanks very much. Thank you. Have a nice day. Thank you. Bye.
[80:01] Thank you. Bye
[80:04] a good day, night, whatever it is. Thanks. Thanks, Annie. Thank you. Bye.
[80:09] Thanks, Annie. Bye.