cscott/ask-annie
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

batch transcripts

Browse Source
This commit is contained in:
Conan Scott
2026-03-24 20:47:49 +11:00
parent 6b3a2e739a
commit 9ec1517274
62 changed files with 3828662 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

987
out/981433083/transcript.txt Normal file
View File

@@ -0,0 +1,987 @@
# Transcript: 981433083
# URL: https://vimeo.com/981433083
# Duration: 4713s (78.6 min)
[0:01] We are not recording yet. How's give us okay. Recording is on. Yep. Go ahead.
[0:07] Yeah. Okay. We are creating an account
[0:12] with the with the rest API and the subscription
[0:15] and rules.
[0:17] And,
[0:18] if I do the subscription, I don't see immediately
[0:21] the directory
[0:22] on the server.
[0:24] Yeah. I
[0:26] found out the first time the logging
[0:28] is done
[0:30] by the customer, then the directory will be created.
[0:34] Yep. It's either so subscriptions
[0:38] will be created either when a user logs in to initiate them, or
[0:44] alternatively,
[0:45] if there is a pull
[0:49] scheduler,
[0:50] if any type or folder monitor, we will create it the first time we need to put the file into the folder.
[0:57] So you are not missing a flag or anything, it's just how it works.
[1:02] When
[1:03] a user logs in, we will check
[1:07] if all of their subscriptions
[1:09] are existing on the file system, and we'll create the missing ones.
[1:13] That's how it's supposed to work. And
[1:16] yeah.
[1:17] If if if if I put a file from the inside to the outside in out directory,
[1:23] So the out directory be created as well before
[1:27] before the the user
[1:30] has done the first login.
[1:33] How are you putting the file in? With a pull?
[1:37] Yeah. With a to to to Yeah. No. From the inside with with a with a root.
[1:42] Yes.
[1:43] If you are so if you are doing publish to account, we will create the directory as well. Yes. Okay. Okay. So that's So if so when ST is knows that there is a file coming with a pull or publish to account, we will create what we need for it.
[2:00] If we are having
[2:02] if a user logs in, we'll also create everything. So the only corner use case is if you're doing something weird like putting files into the folder from the OS level,
[2:12] which please don't do that. But if you're doing that,
[2:15] then the folder will be missing because we don't know yet we need it.
[2:19] Other from that, if ST is delivering, you don't need to worry about that.
[2:25] And if you hold on a second. Let me just go very quickly on my server because I think we actually had the control for that somewhere.
[2:32] I see raised hands will go in order. So just give me a second, guys.
[2:38] And you're not seeing my screen, are you? Hold on. I forgot.
[2:41] Let me share my screen.
[2:44] Yeah. Not yet.
[2:45] Coming.
[2:46] Let me know when you see it.
[2:48] We see it.
[2:55] So over here on the publish to account, whatever folder you specify,
[3:00] because it's folder to folder and we know it is, we will create this folder if it doesn't exist
[3:06] always. Okay.
[3:08] Okay. So you don't need to worry about it. It doesn't even so if you don't want to be created, see, we have a disable over here. The default is we always create. However,
[3:19] if you don't want to be created because of typos and so on, if you click this checkbox on the published work out, it will not create a target folder.
[3:29] Okay. Super. Yeah.
[3:31] And when we say target folder, that also includes a home folder. So if the home folder is missing, we'll first create a home folder and everything and so forth. It's it's just we'll we'll make sure we have a place to put the file. So that that's so you don't need to worry about that.
[3:46] Yeah. Okay. Thank you for the answer.
[3:49] By the way, we, I have a customer that actually like that feature because it allows them to very easily find out how many users had never received the file yet.
[4:00] Because if they it never had logged in. Because if an account logs in, they will see the folder. They it will create it. If we publish an account file for them, they will actually have the folder. But if someone still don't have the folder, that means they were never around and there were never files for them.
[4:17] So
[4:20] you know? Yep. Yep.
[4:22] Okay. Thank you. Okay.
[4:24] Absolutely.
[4:26] Jort. Okay. I probably messed up your name.
[4:32] That's pretty close.
[4:34] I'm trying. I'm European. I'm trying.
[4:39] Alright. Well, I have a question maybe you haven't heard
[4:42] very much of, or at least I haven't heard very much of, relation to ST's external scripting.
[4:49] Yay.
[4:50] Yay. What I'm trying to do is to develop a sanity check, like, on an outbound file,
[4:57] scan the file, determine if there's any data anomalies.
[5:00] And then if there is, then I want to actually move the file out of the pipeline to sending to the client. So it's like, don't want it to go to a client or a vendor. I wanted to to quarantine that file, and that way we can, you know, have time to look at it later and fix whatever problems there are. So I kinda figured out how,
[5:19] you know, to do general external
[5:22] scripts is is so we're
[5:24] we're hosting
[5:26] ST on Red Hat.
[5:29] And so the external script
[5:31] itself is in Bash,
[5:33] and then the decision making module is in Python. So Bash will call Python,
[5:39] which will process the file and then
[5:43] send some kind of signal, I guess, to to the Bash script, which will pass it on to
[5:48] the next step in in the route.
[5:51] So what would you recommend?
[5:54] Yeah. You cannot really pass anything back besides
[5:58] yes, no. So external scripts are very limited at what they can do. You really cannot pass a lot of information back out because both the Python and the Bash handlers are basically not in ST per se.
[6:12] So the way the easiest way for something like that, and it's just a yes, no. Right? You either say that file is good to go or that file is not.
[6:22] Speaker Correct. Speaker So you have two choices here. So let me ask one more question. When
[6:30] you don't want to process the file, do you want still to move that file somewhere else, or do you just want to
[6:38] stop doing anything at this point?
[6:41] Well, I don't necessarily
[6:43] need for ST to move the file. I mean, it it automatically
[6:48] goes into it it well, I just wanna clarify. If if ST is moving in into
[6:53] a location
[6:55] to be scanned, right, a sandbox,
[6:57] then that file is in the sandbox.
[7:00] Should I move that file out of the sandbox using the Python or the or the JavaScript?
[7:05] No.
[7:07] Here here is what why I'm asking. So if you have an external step,
[7:12] let me you can see my screen. Right? Yes.
[7:15] So basically, do whatever you want here in the script. The end result from this step will be either a yes or no, success or failure.
[7:23] If you do zero, it will be a success. If you do a minus one, whatever, it will be a failure. So when you go to the next step from here, and let's imagine it's a publish to account,
[7:34] it will either receive a success or a failure based on what the script would get.
[7:38] And that's where you can communicate with that. All you need to do is to not proceed on exit on failure.
[7:45] And then if you come with the failure from the sandbox,
[7:48] from the previous step, from the external script, it will just not go into the next step.
[7:54] And that's it. That's all we want to do.
[7:57] Okay. Well, if we're not moving the file out of the sandbox,
[8:02] and then we wanna fix the file and send the exact same file name and everything's different except for maybe the timestamp on it, We send it back through. Won't that file that's in the sandbox interfere with it, or is it automatically self cleaning after?
[8:15] So the the moment the routing finishes for this file, destroy the sandbox immediately. So the moment you exit the route, the sandbox is gone. Nothing will interfere anymore.
[8:25] So you don't have to worry at all. And the reason I ask what you want to do is because your other option,
[8:31] it might be even easier than that. So
[8:35] If you find that the file is not good,
[8:38] you can just delete it from the Sandbox so it disappears from the Sandbox. It will still be out in the subscription folder, so you still have the file outside. That's why I ask you if you want to deliver it. Because if you want to deliver it from the Sandbox out
[8:51] to somewhere else, then you can do one proceed on failure, one on success, and you can do both of them and on failure send it somewhere else.
[8:59] But if you all you want is to discard it,
[9:02] you can just delete in your script so that on the next step, and don't forget here to uncheck this after an external script because external script cannot pass files,
[9:11] so always uncheck that. So what it will happen is when you go to the next step,
[9:16] and if this is unchecked,
[9:18] the process only results from the preceding step, it will just look at what is still in the sandbox.
[9:23] And if you have deleted the file from the sandbox, there is nothing. It will say, don't have files to process. It will exit. Always done. We're done with that.
[9:31] So there's two ways to do it. How you want to. Yeah. Yes. Yeah. So I can just check the process only the result from the preceding step. And if it returns a one a minus one,
[9:41] then it just will ignore the file that's in the sandbox. So it's that's the same result. Right? Yeah. Yeah. You can you probably we can probably find out three more ways
[9:51] on top of that. There is a lot of ways to do it.
[9:54] No.
[9:54] ST is very flexible.
[9:57] So I'm just get I'm just breaking into the, you you know, the whole thing. I've I've written a couple of things that we're using in production right now, external script wise, but Yeah. This is by far the most common or it's going to become the most common type of sanity check process. Yeah. Yeah, don't forget now you also have the condition over here where you also can see the preceding step exit status, see if it is success, sex, or failure as opposed to going with the proceed. Sorry.
[10:22] I misspoke, and I apologize. The proceed over here is actually for what to happen after this step. For what we're talking, you need this this expression over here on the top. I'm sorry. My brain just
[10:34] hit a little bit of a problem.
[10:37] So what you actually want is to do this.
[10:40] If it is a success, only then continue.
[10:44] Okay.
[10:45] So that's the step that needs to happen. So that that guarantees that nothing in the sandbox will go if the preceding step has returned a minus one or whatever Yes. Or failure. Yes. I'm sorry. I was on the wrong side of the step. Because Oh. This at the bottom is for
[11:03] for what to happen after this step. And external scripts don't have disconnect disability because they always get out, basically.
[11:13] But this new expression is how you want to do it.
[11:17] Okay. Well, thank you very much. That that helps a a great deal. I'm sorry for the extended time on the on the question. Oh, no. That's okay. I I just went the wrong way and my brain just just caught up with me.
[11:29] But yes,
[11:31] if you ask me what I would do, I would actually delete the file from the sandbox from your script.
[11:37] I like cleaning up things.
[11:40] But part of this is because this part over here is new. We just added it about a year ago.
[11:46] While the other way with the deleting of the file had existed since advanced routing had existed.
[11:51] So
[11:52] I might do both just for extra measure of security.
[11:56] Yeah. You can. You always can. And I and I actually might take the to use the Python piece to just, like, say, if this file is good, hey. Let it go on. If it's bad, hey.
[12:07] Copy it to an error directory and then delete it. And then that way, you know, just pass on the minus one to the
[12:15] the calling bash file.
[12:17] So You can do that. Keep in mind that just that way, this copy will not be visible in tracking table and so on. And perform depending on the size of the file, the performance can suffer because we have now an invisible
[12:31] transfer that is adding to your timelines
[12:34] that you might need. So I don't like using scripts to move files around. So if you want to move it, what you can do is leave it there, do one step which is on success, do this, and do another step after that, which is on failure,
[12:49] do that. Yeah. That and then use a publish to account or send to server or something else to move the file wherever. That's why I asked you if we want to move the file because that's where I was going with that. Because if you So that that adds transparency
[13:04] to the process. That way, it's auditable.
[13:07] Yes. And and it's not a black box. Okay. Great. Great. Great. Thank you. And you can learn reporting, and you can get you can see who is sending you wrong files, and you can go scream at them and tell them, guys, every second of your files is going into processing.
[13:22] Can you Oh, that's always my bigger part of the day is when I can go yell at somebody for something like that.
[13:28] Hey. Why are you messing up the files? Yeah. And you will not have and you will not have other people screaming at you because their files are moving slow because you need to copy them three times around in the script and you don't have visibility into it. So Yeah. That's true.
[13:44] Okay.
[13:46] Okay. Where did my ping go? Okay.
[13:51] Okay. I will mess up another name now. Gavaskar?
[13:55] Yeah. Hi, honey. Yeah. You can call me Gav.
[13:58] Okay. It's easier. Hey.
[14:02] My
[14:03] question is regarding ad hoc transfer.
[14:05] So currently,
[14:07] for any ad hoc transfer, we are using kind of a account. We create, like, account template,
[14:13] and then
[14:15] we provide,
[14:16] you know, access to that account template to users
[14:20] to do the ad hoc transfers.
[14:23] Is there any better way to do this? I see there's some ad hoc settings which I've
[14:28] gone in there, but I don't quite understand.
[14:31] So is there any quicker way
[14:34] to
[14:35] do ad hoc transfers, you know, by using,
[14:38] I don't know, email addresses or temporary
[14:44] users and things like that? So in order to be doing ad hoc, you either need to have a account
[14:51] or
[14:52] you can map into a template at real time.
[14:57] So the ST template accounts can be used for two separate things. One of them is onboarding so that we create an account for the people,
[15:07] but they also can be used
[15:09] for people to log into them. So what are you using, actually? Are you creating accounts for everyone, or are you just using the standing template?
[15:18] So we we're using LDAP for authentication. So we we don't create accounts for users.
[15:26] So
[15:27] we create the account template and then, you know, give the LDAP account
[15:31] access
[15:33] to that,
[15:35] you know, user class account template like that.
[15:38] Just a single template or or so do you have how many accounts?
[15:43] How many templates do have?
[15:45] For ad hoc, we have five. So we kind of rotate between,
[15:49] you know, we have five available ad hoc accounts, then we kind of use them
[15:55] as and when it's required.
[16:00] I'm not sure what the question is, Gav. I'm sorry. Because Is there a better way to do this ad hoc No. Than
[16:09] You either need a template, and when you have l dApp, that's the clean way,
[16:14] or you need an account. You just we we need an object in ST to hold the credentials
[16:20] or the rules for it.
[16:23] So
[16:24] with template, you already have the better option.
[16:28] So as long as you're not creating experience yeah.
[16:32] Because I've seen setups, there's some ad hoc settings.
[16:36] When I go in there, I don't quite understand what the settings We can go through them.
[16:42] So
[16:43] I can go through them very quickly.
[16:45] Right? So
[16:47] the
[16:48] manager base over here is where all the files will go. So the way it works with ad hoc is that we don't keep the packages inside of the user's folders. The user's folders only have links,
[16:59] and instead, all of them will be in this folder.
[17:03] So it doesn't make using account when we're using account template,
[17:06] it has its own home directory. Right? So what is this? And no. No. No. No. Okay. So
[17:12] the template and account home directory has a subfolder
[17:17] called underscore mailbox usually. This is where you specify the name of the folder
[17:22] where there will be links to the packages that this user can use,
[17:26] but all of the data files are in the base folder.
[17:31] So in the regular SD server, when you upload the file for user gulf, it goes into the folder that belongs to GAF. That's how the FTPSSH
[17:40] servers work. Right?
[17:42] With ad hoc,
[17:44] when AMI sends a message to GAF with a file or without the file, it might be just a message,
[17:51] the file
[17:52] doesn't go into the GAF folder. Instead, it goes to the package into this base folder
[18:00] and get linked
[18:02] into Annie and GAF's folder. So there is one package with two access points.
[18:07] This is for ad hoc messages.
[18:09] If you are asking about sharing files, that's different. So which part of ad hoc are you using mostly?
[18:15] Is it messages or files?
[18:18] Early files. We only use file transfers.
[18:22] Okay. So so you
[18:24] don't
[18:25] use packages where you can type things and so on. You're just using sharing files. Right?
[18:32] Correct. Yes. So in this case, this part is irrelevant for you. This is for the other part of ad
[18:38] So ad hoc in SD has two features. One of them is like a mail like system
[18:44] where you can send the package, and we can save it in the package folder and so on. But when we're talking about files, shared files, what you are doing,
[18:53] all they are doing is we just basically create the links on the fly in the folders, in the STFS folders actually,
[19:01] and that's it. So that part over here is irrelevant for you. We never use it.
[19:07] Oh, okay. Alright.
[19:09] And that's why they don't make sense to you because
[19:13] they belong to the other system.
[19:17] Okay. Alright.
[19:19] I thought this is just for
[19:22] secure transport is for file transfer. I don't know I don't know what messaging and things work. So okay. Anyway, that's for another time.
[19:29] It's it's just it's just another way to send files. So it allows you to basically send a message to someone that contains an attachment or is just a message saying, hey. Where is my file?
[19:41] And they will receive a notification and then can go into ST. There is a mailbox
[19:46] alongside.
[19:47] If it is enabled and it's not on my server, or I will log in and show you. But if it was enabled, you'll see it. It's on the web client. It basically opens a second UI over there, a second tab,
[19:58] where you can it's almost like a mail
[20:01] kinda thing.
[20:02] You can have attachments, and we can enroll people.
[20:06] So if anything, the shared fault files that you are using are actually the secondary feature to that one.
[20:13] We just tie them together at one point.
[20:16] Okay.
[20:17] Alright. But that's why
[20:19] I think that's why it's a little confusing for you because whenever you say ad hoc, everyone thinks messages, and that's where I went as well, obviously.
[20:28] So whenever you ask support or anyone about your ad hoc mention that it is file sharing and not ad hoc itself
[20:37] so that they know Okay. What what to explain. And that's why this whole halting doesn't make sense because
[20:43] it's So how are you how are you doing it now then is is the only way. Right? Account templates and things. Yes. You're actually doing it the better way.
[20:53] Okay.
[20:53] Alright. That's fine.
[20:55] Alright. Thank you for that. I got a couple of more questions, but I'll ask after the next question. I don't wanna wait. Okay. Well, we have Mark Lee, and then I'll come back to you.
[21:05] Alright. Oh, hi.
[21:07] Hi. Thank you for taking my taking my question.
[21:10] So I'm pasting these in the chat session. So there's two error messages that I'm seeing in the server log. I was just wondering,
[21:17] generally speaking, what can I do to resolve these errors?
[21:22] So the first one, protocol commands, batch size,
[21:27] you basically
[21:29] will need probably need to increase it.
[21:32] It is let's see.
[21:36] Let's see if I can find it quickly for you. So for the first one,
[21:42] if you had never worked with that, you can always call our support.
[21:48] Okay. I'm not sharing my screen, am I? Hold on.
[21:53] If you are transported.
[21:55] So
[21:57] the first one is basically this setting in the server configuration.
[22:02] Okay.
[22:03] You can increase that one a little bit that the fact that it's complaining that for performance that tells me that your server was a little busy.
[22:14] Are you enterprise cluster or standard cluster?
[22:17] Standard cluster.
[22:19] Yeah. Open a ticket to support about that one. Because on standard cluster, if you increase the batch too much,
[22:26] it will degrade in a different way.
[22:29] So you can talk to them to see what value. But, basically, what is happening is that you have way too many protocol commands that need to be written into the database at the same time,
[22:39] and the value is too small, so they get hanged up. And that's why the server is telling you, by the way, you have way too many of those things.
[22:47] So you might want to increase because, for example, you have a thousand of them and this mean 10 batches. If you increase to 200, it will be five batches, so it'll go faster.
[22:57] But
[22:58] depending on how often you see it,
[23:00] that might be that detrimental.
[23:03] Because if you increase it too much
[23:06] and you don't have load all the time, everything else will might slow a little bit. So it's this fine tuning part of it. And that's why I asked what database you're using because with enterprise clusters, usually, you have a lot more data.
[23:19] So it's more likely for that to happen. But that's that's what it is all about. It's basically pointing you to that one. You might want to check that one. If it's not a 100, if you had changed that,
[23:29] set it back to 100, restart, and see what happens if it stops. If it is already 100, get support, see what their advice will be.
[23:37] Okay. Got it. Okay. And the second one,
[23:40] file archiving retention period is not an integer.
[23:44] Oh,
[23:45] you have a misconfiguration
[23:47] somewhere.
[23:48] So it might be
[23:52] server again.
[23:54] So under setup,
[24:00] there is multiple places where this might be coming from.
[24:05] My my guess is it's actually coming from the
[24:10] account main file maintenance application,
[24:14] which
[24:15] have too many application.
[24:18] Not
[24:20] the account maintenance.
[24:28] Oh, well, let me just create it. I might not have it.
[24:33] So there is the file maintenance application
[24:37] that determines
[24:38] how old the days need to be before you delete them.
[24:43] Mhmm. And it's possible that you have a weird value over here. So the good news is that it's just a warning, so it will go with I'm not going to delete anything.
[24:53] So maybe
[24:54] you put a too big of a value,
[24:57] and it doesn't recognize it as an integer. Maybe someone mistyped
[25:01] something, can put a commas there or dot or something like that. And don't forget that this also can be overwritten on the account levels as well, so it might be somewhere else. So start with the application and chase it down and see if you can find it,
[25:15] but it's just a misconfiguration.
[25:17] If you look at the message, it basically tells you because I don't know what to do, I'll use minus one, which means don't clean anything.
[25:25] Mhmm.
[25:27] So that's Now
[25:29] this one, even though it can be set in a different in a few places, actually,
[25:34] I think let me just check quickly. So we're configuration.
[25:39] Yep. Hold on. Share. Share. Share.
[25:43] I keep me sharing. So
[25:45] so over here, it's also over here. You cannot edit from here because it's in a different pace usually,
[25:52] but start from here. My guess is you'll see something weird here instead of the server. Maybe it's empty. Maybe it's it's just a misconfiguration
[26:01] of the server, and most likely you just mistyped something.
[26:04] I'm not going to tell especially
[26:06] because in a lot of those fields, we don't have control and there is no checks and balances, so it will allow to save pretty much anything or maybe someone used an API. I don't know. But that's what it is all about. So from the two errors you see,
[26:21] if you don't want to do file retentions anyway and you rely on it, I would just ignore the second until you find it and see where it's mistyped.
[26:30] The first one,
[26:32] see if it's 100 at the moment, if it is not restored to 100 unless support told you to lower it and just see with them what they'll say.
[26:42] And how often do you get the first error?
[26:46] Every day.
[26:47] Once per day? Or There's a I I see multiple
[26:52] entries of this. I check the logs every day and I see I've been seeing it for the past
[26:57] week or so. So
[27:00] yeah. Yeah. That's that's why My
[27:03] my guess is that
[27:04] you either have a little bit more volume this week or someone changed something they shouldn't have or something like that. So start with checking the value, and if it if it's not changed, get with support to see where to increase it or
[27:18] increase it to 150, see what happens. You know? But it's a production system.
[27:23] Alright? And I don't like changing things on the production system without checking things first. So Right.
[27:29] Okay. Yeah.
[27:30] But it's
[27:31] we have been trying to to get those
[27:35] messages better. So when you read them, we actually tell you what to do. And when you see by the way, whenever you see something like that that says value of and then some name with dot in it, always grab the copy this thing and put it in server configuration. You saw where I went, right, on the server configuration because 90% of the cases, it's one of those. If it is not there, then it is one of the configuration files. Right? Like the
[28:02] pulls of the database and so on. But when it's one of those, and then if it is editable from there, obvious what you can do. If it is not, then there is another page somewhere on the server from where you're editing it,
[28:14] like the second one.
[28:17] Okay. Alright. That was very helpful. Thank you. Okay.
[28:21] Gav, we'll get back to you in a second.
[28:24] I see that you raised hand. So, Arpit?
[28:27] Yeah. Hi.
[28:29] So we have a new use case that for all of the user initiated
[28:35] uploads, we
[28:37] want to have
[28:38] a total
[28:40] volume limitation or a size limitation of the folder that they are subscribed to. So is there a way that we can do it at the account level or at the subscription level to limit the amount of storage
[28:50] that they have for their folder by any chance?
[28:54] No. Unfortunately unfortunately,
[28:57] we don't have a good way to do it because we are multithreaded.
[29:01] Mhmm. It's not even easy to implement
[29:04] even if you want to build something. So
[29:07] for the most part, the we
[29:11] so there is a idea in the portal you might in the ideas portal. You might want to vote for that. I also know that they had been looking to see what they can do later this year or next year about that. So there might be something coming, but it's not committed yet.
[29:26] Okay. Not completely.
[29:28] But the the way the moment the server stands at the moment,
[29:33] we do not have
[29:35] any capability
[29:36] to restrict people how much data they can send us. We can restrict the bandwidth they're using.
[29:43] But that's about it.
[29:45] Understood.
[29:46] Thank you. The
[29:48] way to do it, by the way, if you want to do something outside from ST,
[29:52] if you have Sentinel or any other monitoring or
[29:55] you have folder based monitoring on OS level. And the moment when
[30:00] you reach the quota, you can disable the account or lock them or something like that. You know, it's ugly, but that's pretty much all the options. Understood.
[30:08] Send send a warning at at at a certain threshold Yeah. Then maybe disable or block the account. Yeah. Understand. And you can do that either through the OS level monitoring if you are receiving but not deleting. But and it also depends on do you care how much data is received or how much is saved? If you're looking for the received, you'll need to calculate from outside, either through the APIs or if you have Sentinel, I will do a correlation a correlation rule in Sentinel.
[30:35] That would be my solution, and that's what I usually built when I have this use case. And then I just sent a note, this guy already sent 50 gigabytes today. Someone needs to talk to them, and then an admin can decide what to do. Understood. But nothing kind of dramatic, I'm afraid. Okay. No worries. Yeah. And and as I said, they are in there looking into what to do, but the server is just built for get us as much data as fast as possible. So it's kind of counterintuitive,
[31:02] but we have this request quite often.
[31:04] And that's why, as I said, go to the ideas portal. There is an idea for that. I'm I'm sure because I I know a few customers that already put it in. And
[31:15] yeah. Mention your use case. Yep. Definitely.
[31:18] Definitely. So I just have another
[31:21] question actually that's
[31:23] a follow-up to one of the previous
[31:25] Sure. Sessions I attended with you. So I talked about the partition days to prebuild option that's available in our server configuration now. Yep. And we use it for transfer log maintenance.
[31:37] So in in our particular use case, we run the transfer log maintenance every
[31:41] seven days.
[31:43] When I say weekly once, actually, on on Saturdays,
[31:46] instead of doing it daily as recommended by by Xway. That's that's the history around it that since in the older
[31:52] versions of SecureTransfer when we were running,
[31:57] we
[31:58] did not have the partition partition capability within within Xway
[32:02] to
[32:03] so we just had one partition and to clear it out every day used to hamper the file transfers around that time. So then we moved that to happen on the weekday, that's
[32:13] weekends, and that's the same setting we followed onto the new implementation five dot five. And now we got to know there's a partitioning capability available that we can use use and
[32:23] and then do it. So coming to the question
[32:26] so in in the last session, we spoke about how the partitioning happens, and then then you showed there is setting
[32:32] or the time at which the partitioning happens once we set that setting, for example, say twenty or twenty five days.
[32:40] And then it creates that many number of partitions.
[32:43] And based upon the transfer log maintenance, it should drop as many number of partitions as possible
[32:49] based upon the days we we have set in in that setting. So when when speaking with the with the support on on that which we're
[32:57] working with them to implement this in our lower environment, and then we'll do it in the the production environment.
[33:02] So they explained that
[33:04] the number of partitions that it creates for the days that depends upon when you run your transfer log maintenance.
[33:10] So that if you're running it every seven days,
[33:13] so one partition will still hold the data of of of of of seven days worth and there won't be seven partition that it drops the older partition. So how
[33:23] would you explain? Because this contradicts
[33:26] with what we discussed in the last session or I may have understood it incorrectly.
[33:30] So I just wanted to bring it up again and and and and
[33:34] and then see what what you explained to that.
[33:37] Honestly,
[33:38] I here the answer will be, I don't know. I'll need to come back to you because I
[33:43] had never
[33:45] run
[33:45] that application
[33:47] not running every day. Mhmm. I I will play with how many days I'll keep, but I will always run it daily.
[33:55] On a daily basis. So that's why my explanation on the partitioning and so on had always been the daily one. So
[34:04] I don't know if when we're doing seven days, if the partitions are per period or are they
[34:12] per day.
[34:13] My understanding
[34:15] historically
[34:15] was that we're doing the daily.
[34:18] But, again, I'm not sure if it is because we're doing the daily or because I always had had it set on the daily because that's always had been our recommendation,
[34:26] and that's how it works and because it makes it easier to do the with Oracle, especially when you talk Oracle to do based on days because then the searches are easier inside of a day. Right? So
[34:39] I will need to go back and talk to r and d and
[34:42] support and see where we're standing on that. It's an unusual use case to say the least. Yeah. Definitely. And actually, we are trying to go back to the usual one to run it on a daily basis. But since the situation we are in, we want to assess it and then and then get it rightfully done in the lower environment that you're working with the support anyways.
[35:00] So Yeah. Hopefully, we are able to find the answers to that. Okay. Can you yeah. Can you send me a mail
[35:08] just to remind me not to I'm I will not forget. I took a note, and I'll try to chase down the answer.
[35:15] Okay. I think
[35:17] I I know what they are saying, and I understand where you're coming from. Because if we're if if you're running every seven days and we're building the partitions based on the seven day window as opposed to the one day window,
[35:29] if you change the window, what happens with the old partitions and how we'll handle all that. Right? That's the two questions here. Right? So I
[35:36] and I don't know the answer to that,
[35:39] but I will try to chase it down. Because as I said, usually,
[35:44] what we will do in
[35:46] what I'll do in other customers and what I had done historically, I'll play with the number of days that we keep, but I'll never change the running schedule. The running schedule is always daily for me. Mhmm.
[35:59] So
[36:00] okay. I will try to chase it down for you. That's I like questions I don't know the answers for. Believe it or not, it actually gives me something to play with. Okay. I'll try to figure out what's gonna come. Thank you. Yep.
[36:13] Okay. As I said, I don't know everything.
[36:16] Okay.
[36:17] Antoine?
[36:20] Hey. Hi. We
[36:21] have
[36:22] a
[36:23] customer connecting to us, and we wanna restrict
[36:27] the folders that they see.
[36:29] And we tried with the aspect restriction,
[36:32] and then
[36:33] we're I don't know if we're not setting up right or something. Like, could you talk to that a bit? So
[36:40] when they log in, you want to risk that you don't want them to see all the folder in their home folders?
[36:46] Correct.
[36:48] Yeah. Or or they could see or they could see it but not be able to access it.
[36:53] And when you say access, you mean not able to upload, not able to download, or not able to even open?
[37:00] All three.
[37:01] All three. Okay.
[37:04] So the first question, can we restrict what they see? The answer is no. We basically will show them everything that is in their home folder.
[37:12] Okay.
[37:14] It's we there is no extensibility.
[37:16] You can write an authorization
[37:18] plugin
[37:19] that does something else eventually
[37:22] with the list type, but that's pretty much it. So that's the usual way to do it. You only need to do a pluggable.
[37:28] But once they're in the folder,
[37:35] what you can do is if you go to the restrictions menu,
[37:39] and you see I've been playing Quick Links here,
[37:44] You have when you're doing a
[37:48] for the folders you don't want them to see,
[37:52] you can restrict the access the the access to the file directory,
[37:57] which will mean that it will give them act if you set it up properly, it will give them access denied
[38:03] if they click on it, which is ugly, but that's pretty much all we can do.
[38:09] So
[38:10] you can restrict
[38:11] them, but you cannot
[38:14] what so you what you can
[38:16] are they using SSH or HTTP?
[38:21] Or both?
[38:22] SSH. That's that's better, actually. Because HTTP has a little bit of a wrinkle because of the way we're loading directories there. But what you can we so
[38:33] you will not be able to restrict this. If you restrict them access to directory,
[38:38] that means that they cannot do anything inside of that directory either.
[38:42] And that's
[38:43] kinda logical,
[38:44] but we had that problem with another place. So you will not be able to say, don't allow the directory, but still allow me to download files from inside. That won't be doable.
[38:55] But if you want to restrict them on the directories, you specify the path over here. This is a relative path
[39:01] to the home folder.
[39:03] Okay. And the class, you know how it works. You can have as many classes as you want, obviously.
[39:10] So and you also can do a depending on what you want to do, if you want them only to be able to risk to get
[39:17] into one directory and nothing else,
[39:21] then
[39:22] you can do a blanket. You cannot access anything and then allow them to access.
[39:27] So here is another question for you. Why do they have folders they shouldn't be able to to go into? Just out of curiosity.
[39:35] Well, when they log in, they see, like, the subscription folders,
[39:39] and we don't want them to look at those at all.
[39:45] Okay. And this is SSH?
[39:47] Yes.
[39:49] Okay. I will give you another trick. It won't help. So you still need to do that. But in addition, you might want to also
[39:57] what happens if your subscription folder start with a dot?
[40:03] Because those would be Yeah. We we don't we don't have any with dots, but okay. So please Or you can try. Well, SSH is a little weird that way because it will still allow them to see them if they know what they are doing, but it might stop the
[40:17] some of the.
[40:18] You know? K. I okay. I I see where you're going. Okay. But it doesn't mean you shouldn't do the restrictions. So you still need to come here and do the restrictions.
[40:28] But I also will say if all you want is to you do use those folders to process anyway,
[40:35] you might as well and you don't care what the subscription folder is because you don't because you don't want people to go in, you might as well use the fact that dot folders are hidden by default by most by most clients.
[40:46] So Okay. I would do it both ways. But yes. So what you want to do if you don't give want to give them access is to access file directory from here
[40:55] and just
[40:57] don't allow them to. And I would also make sure that make a directory is also forbidden
[41:03] so they don't create their own directories. That's a generic thing I do on most of my servers
[41:09] unless you want them to be able to create the directory and so on,
[41:13] and that's about it. The other option is if you don't mind them looking into the folder, but you just don't want them to upload download,
[41:20] you can play with upload download restrictions instead.
[41:24] So Yeah. The thing the thing that was happening is they were putting the files where we wanted we're expecting it, and they were putting in this subscription folder.
[41:31] So nothing was triggering because the file was not supposed to be there. So
[41:36] okay.
[41:37] Okay. Well, if that's your problem, I would probably do upload restrictions instead of the file system restrictions just because
[41:44] it looks uglier when when they do CD and give them permission denied, they'll or or access denied. It will usually come to you and tell you I cannot get into my folder, and you have a hard time explaining,
[41:57] you know, you're not supposed to be there.
[41:59] So up to you, but you can either do access from here or just put upload restrictions on the folders.
[42:08] Yeah. Okay. Thank you. Mhmm. And if you use the dot
[42:12] as a start, that also gives you the ability to do a easier recognition of those folders.
[42:19] You know? Yep. Yeah. I I think we're gonna try that out. Thanks. Yeah. Again,
[42:25] it won't help completely,
[42:26] especially because it's SSH and some of SSH clients are space are are specifically tied to see the dot folders.
[42:33] But I've been doing this historically
[42:35] because especially for
[42:37] less what's the word? Less technical people, they will not even see those folders anymore.
[42:43] But again, do
[42:44] both things
[42:46] as usual. So Okay.
[42:49] Thank you.
[42:50] Mhmm. Okay.
[42:53] And
[42:54] back to Gaf.
[42:58] Yeah. Hi again. Yeah. My questions again
[43:01] is around notification.
[43:03] So
[43:04] we
[43:05] have notification
[43:07] when we route the file, right?
[43:11] But is there any notification that we can set? So
[43:16] when
[43:17] there's
[43:18] a failure to pull the file in the first place?
[43:22] Is there any notification?
[43:28] There it is. Let me get on the server.
[43:33] So
[43:36] when you go to your
[43:39] subscription
[43:42] Yeah.
[43:44] And
[43:47] it needs to be advanced
[43:49] routing. It cannot be basic app. Yeah. But over here,
[43:54] see where it says
[43:58] on failure.
[44:01] This part over here, smack in the middle of the screen, not at the very bottom. So you have so you have three sections here.
[44:09] So let's ignore it's let's forget about that because so it doesn't matter. So the first section four sections, the triggering the you have the on success,
[44:19] which is if the file
[44:20] goes into routing.
[44:23] Yeah.
[44:26] You have on failure,
[44:28] which is if there is a problem during the arrival of the file, which is usually applicable to both sets and set both client and server. And you have the post download, which is if a user downloads a file, and the bottom is the post routing, which is after at the end. But for your case, if you're doing a pull,
[44:45] so you have couple of places where you can do something. Number one, it depends on the failure. If the failure is we didn't find any files,
[44:55] this is actually a success scenario.
[44:57] But Yeah. What you can do is to execute a route
[45:02] when the remote server returns no files. So this will send you into routing, and then you can do a conditional route that says if the wildcard pool was empty,
[45:12] and we'll go I'll go and show you where to set up that, and then you can do a mail notification at this point.
[45:18] Or if it is an actual failure, we try to pull a file but something went horribly wrong, then it will throw you into this section over here on failure
[45:27] where, again,
[45:29] you can execute a route.
[45:32] And, again, if it was a failure and it didn't find file as files as well,
[45:38] you can also check on that. It it depends on how you had set up your servers. Right?
[45:43] But that's how you set it up.
[45:47] Go
[45:50] ahead. Yes.
[45:51] So
[45:52] what we want notification is for, so
[45:55] we try to connect, there's a connection failure, right? If there's no files,
[46:00] if this connection is successful and there's no file, that is fine, we don't need to know. Okay. Then you- Okay. You should see.
[46:09] Okay. Yeah. In this case, uncheck this one.
[46:13] Uncheck this one because we don't want to check if there is no files.
[46:17] And you just do on failure over here and you send it into routing.
[46:22] And then in your routing package,
[46:28] so you usually have your route which are always and so on. This is the positive case. Right?
[46:34] So you have so what you create is a new route
[46:37] that runs
[46:38] on,
[46:41] and here the oops. The trigger will be a transfer trigger pull and transfer status is failure.
[46:49] So there was a server pull,
[46:51] but it failed.
[46:53] See? Okay.
[46:54] You do that,
[46:57] and then over here you notify
[47:00] on
[47:02] route triggering,
[47:04] and that's it. And you save that one.
[47:07] Yeah. You
[47:08] know, I'll just put some
[47:15] that's
[47:17] the wrong one. Where is my failure one?
[47:19] Routing failed. Let's say it like that. And save. I didn't put a name.
[47:30] And save.
[47:34] So now you have two of them.
[47:37] And if all matching, this will always run. This is not what you want. You switch that to first matching.
[47:43] For example, you reorder them. You put that one first,
[47:48] and then you save. So if it comes with a failure, it will hit on this one. If it comes as a success, it will come from this one. Or if you want to do all matching, you'll need to make this one conditional as well.
[47:59] K.
[48:00] That's it.
[48:03] In
[48:05] the subscription, need to select any failed or something?
[48:09] Yes. Then on the subscription, let me just save that one so we don't close it. Then on the subscription itself,
[48:16] you need to send the oops. I clicked on the wrong thing.
[48:21] On the subscription itself,
[48:23] you watch it. So by default, arriving files go into routing automatically,
[48:28] but you also control what else goes into routing.
[48:31] And in this case, you want on failure over here to go into routing.
[48:37] Okay.
[48:38] But
[48:39] how do you select the any
[48:43] failed route?
[48:47] It's already preselected.
[48:48] It's basically Okay. It's all within the within the route. Okay. Sorry. Yeah. Yeah. Yeah. It's always the same package. That's one of that's why you need a secondary route inside of the same package. So at the top, when you select the package that that you use for the routing of the positive files,
[49:04] you just will need to have a route inside of the same inside of the same package. You cannot select a different one. So anything you want to do, any of those checkboxes and sending things into routing from here will be you'll need to be inside of the same package.
[49:20] Okay. Understood.
[49:21] That's why you want it conditional.
[49:23] And you can play with that because there is a lot of other things. You have the post download stuff as well. So for example, if a user downloads a file, you can do something as well, like, someone. Oh, someone picked my file. Yay.
[49:35] We find that I find that very useful, for example, if I do publish to account
[49:40] so that I can then notify someone that the recipient actually picked up the file.
[49:45] But it's just usually, people, when they think about this page, they think about the bottom, the post routing after you came out of the route. But before that, ahead of that, there is a lot of settings that allow you to do things if something goes weird, including,
[49:59] you know, no files found or errors or successes or anything like that. And then you can once you're in a route, you can set mail notification,
[50:08] but you also can run a route that basically calls something. You know? There is no file, so there is nothing to push out,
[50:15] but you can call an external script or
[50:18] something.
[50:20] So Okay.
[50:21] Yeah. That's good. Thank you. Yep. So just play with that a little bit.
[50:26] It's it's pretty straightforward.
[50:29] It's actually working nice. The biggest
[50:32] challenge I think people see with that is what they are forgetting the order of the routes over here.
[50:38] So just make sure that your positive route doesn't run on errors because then it gets a little weird because it's erroring out because there is no file. So it's like, I don't have a file to work on. What are you doing here?
[50:49] So
[50:51] but
[50:52] it's up to you how to handle it. Now I would usually either do a force matching,
[50:57] which always will do the trick.
[50:59] But because there is so many other possibilities
[51:02] in this case,
[51:03] I actually will do this one conditional as well. I will just do it
[51:08] on
[51:13] just on success. Transfer status is success.
[51:17] Oh, and by the way, for the empty files, this is the condition. If you want to catch if you have empty files, if you're expected
[51:23] to have files, but there aren't any, You can check if it was a wildcard pool, if it was empty or not empty, you know, the other checkbox over there. But here, what you can do is on the positive one is to do something like that. Oops.
[51:37] Just transfer
[51:39] status, it will success.
[51:41] This way, if any of the error conditions come in, it will not trigger on that one in case you forget to create one, but it's up to you.
[51:50] Okay. Yeah. Understood. Thank you.
[51:53] Yep. All good. Thanks. K. And don't forget that a lot of that also can be on the steps level.
[52:00] So there is also the possibility to run a single route, and then on the steps, just run them if it was a success. But that's just ugly and very hard to troubleshoot. I find the setting kit up that way
[52:12] with a separate route that
[52:15] that does whatever and naming it properly. Don't call it any failed, obviously. But it's easier to find for me. I write a description and so on, and that's about it. And, also,
[52:26] don't forget you still have the template routes. So if it turns out that you want to do that for every one of your routes, you can also just stick it on the template instead of creating it inside.
[52:38] So it doesn't matter if it is on the template or What what do you mean template prior to what is that? So
[52:44] anytime when you create a new package,
[52:47] you know, when you start building
[52:49] route,
[52:51] you need to select a template at the top, right, which is what is the basics.
[52:55] So most people, a lot of people just use an empty template.
[52:59] But you can have routes inside of your template as well, and they will both work together with the ones from the local.
[53:06] So if you have things that need to be enabled for everyone and usable for everyone, instead of building them inside of the account, you can build them in the template route,
[53:16] and then they will be just coming into the inheritance.
[53:20] Where'd you create the template route?
[53:26] Over
[53:26] here.
[53:30] Those are called template route or route packages or whatever. And you usually have at least one,
[53:37] which is either empty or not. And using templates We have an empty one. Yeah. Yeah. But if you want to so one good thing about the template route is if you have something repeatable like this one, you can put it on the template route itself.
[53:52] And then when you create the package inside of the account,
[53:56] you can actually disable or enable the package level. So if you have let me let me just create one very quickly so can show you what I mean. So if I create a route here,
[54:08] and I won't do anything, I'll just do a empty route just oops. Just so I can show you. So if you have an empty route here,
[54:17] save.
[54:18] Right? When you go to the account to create,
[54:27] my route one zero one,
[54:30] See how it shows up automatically.
[54:34] You can decide if you want it to run or not. So at this point, you can either enable or disable, and that way you don't need to create that one everywhere because it already comes through the inheritance.
[54:45] Okay.
[54:46] And this is a useful thing to do, especially for things like that because then you when you're assigning the route locally, you know, oh, this one will need to have the route the the the checkbox done out on the subscription. So let me keep it enabled, but you can always just disable it. If you disable it, it is as if it never was here.
[55:07] If it's enabled, it will just run, but you don't need to create it inside of every single account.
[55:13] Okay. So if I go and create it now, it will apply to all
[55:16] routes
[55:17] within user
[55:20] accounts and account templates?
[55:22] Yes.
[55:24] Okay. Account template is operational at all times. So if you go and create it in your empty route now, it will automatically
[55:33] populate
[55:35] for
[55:36] inside of all of the preexisting packages that were already created in the accounts.
[55:42] Okay. The
[55:45] default
[55:46] selection will be all matching
[55:48] routes?
[55:50] Yes. But because this one is let me open one of the old ones. See the one we were looking at earlier? See, I changed it, so it now showed up. It the default over here so there are two separate roles rules here because this one is on a separate rule set, and it was when you create it in the template, this is where you specify.
[56:09] And then there is a second point over here. So they can all be all matching and so on. But because it's coming from the template
[56:16] and it's conditional,
[56:18] so mine, I make it always. Obviously, this shouldn't be always, but it was just an example.
[56:23] But it's so if it is enabled or disabled by default, depends on how you set it up in the template. So if you don't want it to be enabled by default, when you go to your template to create it,
[56:34] you can set it as disabled there, and then you can just enable it when needed. So if you want to do it on your current template,
[56:42] what you can do is go to your template. Oops. Sorry.
[56:46] Disable it here.
[56:50] So now it will always show as disabled. So if as if nothing happened, and you can enable it on the local local routes wherever it's needed.
[56:58] Makes sense?
[57:00] Okay. Or you can select first matching route there?
[57:04] Yes. But again, it's a separate list, so there are two of them.
[57:10] But by default, it will be
[57:13] Even if you select even if you select first matching here, this is not the same as on the account because on the account route template on the composite,
[57:22] you have two separate maps. One coming from the inheritance and one coming from
[57:27] the template from the local itself.
[57:31] And with first matching, you have two first matchings. You can have both of them still running even on the first matching. That's what I'm trying to say.
[57:38] Alright.
[57:39] Alright. I'll test it out anyway. Thanks. Yep. But if you already have the template, that's what I would do. I would create it inside of the template in disabled state
[57:49] so that it automatically shows up everywhere, and then I'll go play with an account. Because if it's disabled,
[57:54] it's like it's not there, so nothing will change. But it will be there. And once you know how it works, then you can start enabling by just doing the checkbox
[58:03] on all the accounts where it's needed. So
[58:06] Yeah. Okay.
[58:07] So one thing
[58:09] that is you need to be careful about with that, and you want to test very carefully, is what happens if it coming from the template,
[58:17] but the positive one is on the local.
[58:20] Because
[58:22] in this case, you will need to make the local conditional
[58:25] because otherwise,
[58:27] they both will run on error,
[58:30] which is what you don't need.
[58:31] That's
[58:32] what you always, When you have both template routes and local routes on the composite,
[58:38] you always need to remember that first matching
[58:41] now has two answers. So both of them first matching in both sections will run.
[58:48] Okay. Yep. Thank you. Yeah. And that's why I would just make all of them conditional and be done with that. That's why, you know, earlier how I said that when I do that, I usually will switch the positive on conditional as well just with transfer success.
[59:01] That's the reason.
[59:03] So I don't need to worry who is running. If you are controlling who is running when, you don't worry about always.
[59:10] Makes sense? Yep. Yep. Good. Thank you. Don't play with that. And by the way, for anyone on the call that had never used templates, look at those things, guys. They're very, very powerful, and they gives you a lot of flexibility of what you can do with those servers without needing to do it on every account separately.
[59:29] And when your management ask you to change something, it's a single place to change. But, again, it will change everything for everyone already deployed,
[59:38] so
[59:40] careful.
[59:41] Okay.
[59:44] Okay.
[59:45] Other questions?
[59:46] I don't have any raised hands left.
[59:51] I have a question.
[59:53] Sure. Annie, so for
[59:56] there's a new
[59:58] feature
[59:59] introduced which
[60:00] the subscriptions where we have to
[60:03] do an auto pull from a remote from a transfer site, and there is a call keep pull history for
[60:09] for some days. So how
[60:13] intrusive that setting do you think has has been and how much do you recommend recommend using this? Because we do a lot of pull operations and
[60:20] many a times when the remote server doesn't allow us to do the housekeeping of the file, either delete or rename or move it,
[60:27] we end up downloading the same file again. And
[60:30] this may seem to to be a good feature to use, but just want to
[60:35] check how intrusive that is, how performance intensive it is, and because it will keep the the file names or some some some information somewhere in the database or in cache where it will keep on referring each time the transfer site is is called for or the subscription is called for.
[60:50] Yeah. Actually, it's not in the database. It in it is in the SDFS folders.
[60:54] You know how
[60:56] so it's locally on the file system. So if you have a lot of files,
[61:01] the transfer operations for this specific subscription might slow down a little bit while we are figuring out the list of the files because we need to read and so on. But the way pulls are working is that
[61:13] we get the list of the files
[61:15] from
[61:16] the from the server that says here are the 10,000 files I have. And now we start working one by one to see to to create the pools for them in the database to pull them one by one. Right? That's how ST works. Right. When you enable pool history,
[61:32] what happens is that before we create this small pool thing into the database to pull the individual file, we'll also check, Oh, by the way, do I actually want this file or not? That's
[61:44] all the addition to that. So if you have a huge amount of files in the pull history,
[61:49] that process of deciding which files to pull may be a little slower
[61:54] because we need to check through the list. Right? And it's a file based list.
[61:58] But
[61:59] once this is done,
[62:00] we don't check anything else. And, obviously, when we pull the file, we'll write into the file to tell it, oops, we got the file and there is a cleanup operation.
[62:09] So
[62:11] I haven't
[62:12] seen performance
[62:13] degradation because of that. I had seen it again slowing down on a huge lists just Mhmm. You know, just because with a long list, it's obvious, but it doesn't influence other customers.
[62:25] It doesn't really slow down the transfer itself. It's just the decision process if we want to file or not.
[62:31] And it happens early on. So and it's but it is but I I also can point out that compared to how much time you're losing at the moment downloading the same files over and over,
[62:44] it will actually speed up your server. Because if you need to grab the same 10 files every hour as opposed to knowing, oh, I don't want them Mhmm. You know, the check if we want the file will run considerably faster than actually getting the file. That I can guarantee you. Understood. So,
[63:02] yeah, it's useful. That's why we added it. But also,
[63:06] as you know, pull history is not for all protocols.
[63:09] We couldn't get it to work for most protocols. So hopefully, it's if it is SSH, obviously, it's there. Yeah. For all. SSH. Yeah. Okay. That's right. That's right. Yes, that's what exactly what it was created for. So you don't need to download
[63:22] all every single file
[63:24] Okay. Every
[63:26] So as you mentioned that it keep it keeps that
[63:29] the history in the SDFS folder. And so generally, what we do is we we download a file, deliver it to wherever it has to go, either publish to account or send to partner and just delete the local copy.
[63:41] So the SFS entry will still be available, right, for the full history to to be to be There is no requirement for the file to physically still be in the subscription folder. We don't check the subscription folder itself. We check our record of it. Now if you delete the subscription folder or the account or move the home folder and not move everything and so on, we might lose the history
[64:02] because it's in the file system. But as long as you don't do something weird like that,
[64:07] we don't care if the file is still there or not. When we go an hour later, we know we pull that file an hour ago. We know that it's not new enough to pull again. We we check the size and the time, you know, so that if it's a newer file with the same name, we'll still grab it because we check all of them. But if it is the same file, same time stamp,
[64:27] same size, same file name, we're just going to say, yep. We got that one. We'll skip it next.
[64:32] We don't care if the subscription file is there.
[64:36] Okay. Okay. That sounds promising
[64:38] actually because we are thinking of starting to test it and then see if how it works for us and then maybe actually start utilizing it in production. So thank you for explaining that. So so one thing you need to set up, as you know, is the number of days to keep. Right. What it what it what it does is essentially for how long to keep the list of the files. If you make this value too big and there is a lot of data, this file will keep growing.
[65:04] The the data, the SDFS data about how what files are in. So if you keep sixty days worth of
[65:11] files and you're running a thousand files per day, that's 60,000 files. Right? Oh, yeah. Or 60,000,000 if you millions. But and that's why you and you'll need to play with that. And the way it works is that if if you keep seven days of records and the same file arrives on day eight,
[65:27] we'll still grab we'll grab it again. Because for us, we never grab that file because we are only keeping seven days.
[65:35] That's true. So yep.
[65:38] And
[65:39] mhmm. Sorry. Please go ahead. Go ahead.
[65:42] No. I was going to say it's pretty useful exactly for the use case you have,
[65:45] which is my partner doesn't allow me to delete or move the file after I grabbed it. So that's why we build it.
[65:52] Understood. Yeah. Thank you. That's promising. And just to reassure what the
[65:57] example that you gave in the beginning
[66:00] about
[66:01] how SD functions to pull the files of sales, say, thousand files available on the remote side and we have set
[66:07] in the in the subscription on the transfer side to pull ten ten transfers parallelly.
[66:12] So for each of the file it downloads, will it will open a new SFTP connection, download the file, and close it. That's what the right understanding is or can it download four files in just one connection? How does that Okay.
[66:25] So
[66:27] the default answer is yes. It will be one file per connection.
[66:30] However,
[66:31] for SFTP,
[66:32] we actually have a new parameter
[66:34] that allows us to reuse the connections.
[66:38] Okay.
[66:39] What is that? So Show
[66:41] me. Yep. Getting there.
[66:46] It's one of our other partners has this concern that we open too many connections in a short period of time to download all the files.
[66:53] Nope. That's not the one. What's the name of this parameter now?
[67:05] I'm drawing a blank.
[67:07] Does anyone on the code remember the name of my parameter?
[67:13] Should do something with reuse
[67:15] by any chance or SSH?
[67:26] Well, that's embarrassing.
[67:33] So there is a parameter.
[67:35] I'll find it if I cannot find it now. That basically says
[67:39] pull. Maybe under pull.
[67:48] Here it is.
[67:49] SSH SIP connection pool enabled.
[67:55] K.
[67:56] Hold on. Let me just pull them.
[67:59] See, I told you I don't know everything, but I usually know
[68:03] how to find it. So SSH connection pool enabled, when you enable that is the that's the reusability of the SSH connections during a poor push operations.
[68:12] And you specify how long to keep the connection. So if this is enabled, and for example, you tell us on the site or on the subscription no more than 10 per,
[68:22] then we'll open the first 10 connections for the first 10 files. But for file number but when we are done with the file, instead of closing the connection, we'll just leave it in the pool. If file 11 comes immediately after that, it can reuse the existing connection to send it out, which can speed up things considerably on a huge amount of files.
[68:42] K. Okay. Understood. But And it is Sorry.
[68:45] Please go on. But it it is harmless if you have a low number of files. So if you only have five files and we never reach the threshold of the pool, then we'll never use anything. They'll just close a little slower
[68:59] because we'll keep them open just in case there are more files coming.
[69:03] But in fifteen seconds or something, so your close operation will be a little slower, but the file transfers will not be negatively
[69:10] impacted performance wise.
[69:12] But if you have a lot of files going in the same direction or coming from the same direction, this will actually save you time because we will not do open and close all the time.
[69:21] Understood.
[69:22] But whatever the number of panel transfers we set in the in the transfer site, so that will still open that many number of Yes. Connections and then the reusability will be applicable in in the in the further Yes. Downloads that we do. Understood.
[69:38] It will be the maximum number because
[69:41] if, for example, we have five files
[69:44] and then the sixth file arrives whilst yours and the one of the five have just finished, we will reuse the connection immediately because it's still in the pool.
[69:53] But if there is enough files to open the 10 connections, we'll still open them. We're just not going to open one. So if you say 10 connections on the transfer side, we will open up to 10 connections.
[70:04] When the new files come in, our first job will be, is there an open connection by any chance for me? If there isn't and
[70:11] there is one still not open from the top from the max, it will open it. But if there is 10 open connections that are moving
[70:19] Mhmm. Then this one will be waiting for one of the others to finish the file and then we'll reuse
[70:25] and so on. So that's how it works. We're not going to just wait for files. If there is a connection connection being able to be open, we'll open it immediately.
[70:33] So what that does is just not to close the connection as fast
[70:37] so that if there are more files to go, it can use reuse it.
[70:43] Makes
[70:44] sense.
[70:45] And but and the reason why I'm explaining it's harmless
[70:49] if there isn't enough files is because it's a server side property. So once we enable it, it will be valid for all the SSH sites, not just for the problematic customer. Right.
[70:59] Alright. I understand. But it's again and what
[71:03] you'll see is a little slower closing,
[71:06] but that's about it. And we actually have some intelligent controls that actually can look into the event queue and see if there are actually files stacked up waiting for that specific place.
[71:16] So they might even close on time. They are working on some of the performance consideration. But in my experience, it's pretty harmless to actually enable it to leave
[71:25] it enabled, and I recommend people that know that they might have a lot of files to keep it enabled. The only reason it comes disabled by default is because it's a new feature and we don't enable new feature. But my recommendation usually is to have it enabled for the most part because it's helpful.
[71:42] Okay. Okay. That's again a promising feature I discovered today. Thank you for that. Yes.
[71:48] By the way, this one and the history, the pull history were created exactly because of those big pulls coming from SAPS
[71:56] and
[71:57] had been things people had been asking forever about. So
[72:01] Mhmm.
[72:03] Okay? Yep. Thank you.
[72:06] Okay.
[72:07] We have about ten more minutes before Nicole tells me that we are overtime. So
[72:12] time for last questions.
[72:20] No?
[72:22] Nicole,
[72:23] we're on time.
[72:26] Okay, everyone.
[72:29] So I'll get back to her because she has some other things to finish the call with. I just want to tell everyone or remind everyone that we'll have an s t APIs meeting in two weeks,
[72:41] and I'll be happy to see most of you or all of you or whoever wants to listen in.
[72:48] And other from then, thanks everyone for joining.
[72:51] Back to Nicole for finishing touches.
[72:53] And if you
[72:55] you know how to reach us through the community portal if you need us,
[72:59] I'll be very happy to see all of you either on the ST API virtual group in two weeks or in August on the next ask Annie.
[73:09] And,
[73:10] Nicole, back to you.
[73:13] Yes. Thank you very much, Annie.
[73:16] Great session again.
[73:19] So, yeah, as usual,
[73:22] I just wanted to
[73:25] share with you and remind you
[73:27] about
[73:29] our
[73:30] community portal
[73:31] and the different
[73:33] assets that you can get from there.
[73:36] So you can post
[73:38] your ideas
[73:39] to enhance
[73:41] the
[73:43] secure transport
[73:44] product,
[73:45] suggest
[73:46] some evolution
[73:47] of it.
[73:49] There are also
[73:50] all the upcoming user groups that are posted there and you have the link
[73:56] there to register.
[73:58] If
[74:00] you missed the email
[74:02] that I'm sending to invite you, you can catch up there.
[74:06] And also, you'll see
[74:08] under the user group entry at the bottom of the page,
[74:12] you'll have all the
[74:16] previous
[74:17] sessions
[74:18] and the
[74:20] link to the
[74:22] article where the recording and the presentation
[74:26] were posted.
[74:27] So
[74:29] you know,
[74:30] You
[74:31] can access from the community portal also to the roadmaps
[74:36] of the product.
[74:39] And you have also
[74:41] the
[74:42] forum where you can post your questions and the community can answer either your peers
[74:49] more and more, this is what happened and it's great,
[74:52] or
[74:54] our experts
[74:55] use
[74:57] and abuse of it.
[75:00] What else do we have? Yeah. You have the, what's new video
[75:05] on YouTube.
[75:07] So don't hesitate to connect there, to,
[75:11] subscribe
[75:13] and follow.
[75:14] Now
[75:16] for
[75:17] each new update
[75:20] on ST that are
[75:22] delivered,
[75:24] there is
[75:25] what's new video
[75:27] where it is presented what you can benefit from this last update.
[75:36] And the platform, the G2 platform,
[75:39] if you want to post your review there
[75:43] about Secure Transport,
[75:46] it helps
[75:47] people that are searching
[75:52] for this
[75:54] kind of product.
[75:57] It helps to know
[75:59] your
[76:01] what you think of the product. So don't hesitate to post.
[76:06] And
[76:07] to encourage you
[76:09] to do so and to encourage
[76:11] and to thank you if you do, it is anonymous,
[76:16] but the process give you a $25
[76:21] card,
[76:25] same, don't hesitate.
[76:27] And with that, as
[76:30] Annie mentioned,
[76:32] our next
[76:33] user group on
[76:36] Secure Transport
[76:37] is planned
[76:39] for the July
[76:41] 23,
[76:42] and it will be about the APIs
[76:46] of ST.
[76:48] And then
[76:51] the next session,
[76:52] the next ask any session
[76:54] is planned for August
[76:56] 22,
[76:57] and there will be one in September also
[77:03] in September 12.
[77:06] And it will continue
[77:08] also
[77:08] later
[77:09] in the year.
[77:11] So
[77:12] I hope you enjoy
[77:15] summer.
[77:16] And if you have some vacations plan,
[77:18] I hope it will be great.
[77:22] But for that,
[77:25] we
[77:27] no. Actually, I won't be hosting
[77:29] the one. I have one week of vacation plan,
[77:34] July
[77:35] 22.
[77:36] So
[77:37] I won't be hosting this one. It will be
[77:40] my colleague.
[77:42] So thank you very much. I hope you enjoyed the session again.
[77:46] Bye bye.
[77:49] Yeah. Bye bye. Thank you, guys. That was a wonderful session as always.
[77:53] Thank you so much. Will you be sending the recording, or do we have to check-in the portal for the recording?
[77:59] No. I will send an email with the link to the article posted on the community. Yep. And it will be also on the portal. So you'll get the mail, and you can get it from the portal. So it's not either or. We do both.
[78:13] Thanks, everyone. Thank you. Bye.