Graham
-
Posts
74 -
Joined
-
Last visited
-
Days Won
1
Content Type
Profiles
Forums
Enhancement Requests
Posts posted by Graham
-
-
@Ben Maddams If you still have them, it would be useful to see what the results were from previous discovery jobs. You can filter the job queue by type and so limit it to just discoveries.
-
@Ben Maddams That's helpful, thanks. 'Auto' will also try to connect via SSH, but as most Windows servers don't have SSH enabled, that failure can be ignored. The WinRM connection issue is the one we see when running the job, so that too is expected in this case. However, the 'Access denied' for DCOM is interesting.
I think the next step is to check the credentials used for the discovery. Have they been changed in Keysafe, or have users/passwords been changed on the target? -
@Ben Maddams You won't need to unmanage the device to run the discovery. There's something strange going on with the connection to the target, and I'm hoping that the discovery results will shed some light on what is happening.
-
@Ben Maddams Can you re-run the discovery of the target, with the Protocol set to Auto? Let's see what that does. It does look like something odd is going on with WinRM, but I'm unable to replicate it here.
A discovery protocol of Auto will try using both DCOM and WinRM, so hopefully DCOM will work. -
@Ben Maddams That indicates that WinRM is working OK on the target. Do you get problems running jobs against other targets?
There's been no SIS software releases recently, so the most likely explanation is that something has changed on your side - we just have to find out what!
Can you also run
winrm id
on the SIS server please, just to check? -
@Ben Maddams In that case, what does
winrm idsay? Again, this is run on the target computer from an elevated prompt.
-
@Ben Maddams Hi Ben
The error code 0x80338005 is coming back from the operating system, specifically the WinRM system. As a first diagnostic, on the machine you are trying to connect to (i.e. the target of the job, not the SIS server itself), can you run, from an elevated command prompt:
winrm qc
and post what the result is?
The ping and DNS checks you mention only apply at the time of device discovery - they are not a "live" check in that sense.-
1
-
-
@Ben Maddams We have now been able to replicate this problem internally, and are currently working on a fix.
-
@Ben Maddams Do you have the latest version of the AD User package installed?
You can check by going into the Installed Packages section within ITOM in Hornbill. Version 25 is the latest.
-
@Ben Maddams That's definitely progress! The job is executing, in that the Powershell script is being processed, and it's the Powershell operation that's failing.
I'll take a look at the package. -
@Ben Maddams In that case, as a final check that the SIS service is listening, can you run
netstat -a | find "11117"
from a command prompt on the machine running the SIS service? I'd expect to see:
TCP 0.0.0.0:11117 <name of server>:0 LISTENING TCP [::]:11117 <name of server>:0 LISTENING
which means the service is listening on that port, and I would expect http://10.5.1.153:11117/ to produce the status page from a browser local to that machine (if there is one installed).
-
@Ben Maddams This does look like some sort of communications issue. From a machine that should be able to talk to the SIS server (this doesn't have to be the DC), can you start a web browser and visit
http://10.5.1.153:11117/
Do you get a status page? -
@Ben Maddams Thanks for that. The fact that the job doesn't do what it's supposed to isn't a problem - running the job on the SIS service, rather than on the DC, was the key diagnostic.
Regarding the "job successful", there are two things in play here:
1) Was ITOM able to process the package?
2) Did the package's processing produce the desired effect?
The "job successful" refers only to the first of these. ITOM was able to connect to the target, start the package processing and received notifications that the processing "happened".
That does all point to there either being a communications issue, or something else is interfering with how the remote executable is being processed on the DC.
Is there any anti-virus on the DC? -
@Ben Maddams This is proving to be a bit of a puzzle!
Can you try two things for me, please
1) Run the job with a target of the machine running the SIS service itself
2) Run the job targettting the DC, but without setting the "RunAs" credentials
Hopefully the results from those will shed some light on what's going on.
One other thought - is there any anti-virus software on the DC that could be seeing a new executable running and terminating it? -
@Ben Maddams Thanks for that. It rules out one thing for me.
This still appears to be a comms issue. Can I ask that you run the job again (just so we know where to look in the logs) and then, on the machine running the SIS service, look in C:\ProgramData\Hornbill\Site Integration Server\log\EspSisService.log
There should be some entries in there tying up with the new job run, and that will hopefully give us a better idea about what's going on. The first entry will be
Processing job: <your job number in here>, type=package -
@Ben MaddamsFor the log you posted at 16:20 yesterday, is that all there is? I would expect to see some further lines after the the last one "2024-03-25 16:13:52Z Logging prologue ...."
-
@Ben Maddams In that case, this is still a communications issue. With the change to the config file, are you now getting just one valid IP listed when you restart the service?
Can you run the same job, but use the SIS service itself as the target? That probably won't work, since the RSAT tools probably aren't installed on the machine running the SIS service, but that won't matter. I'm interested in seeing if the job runs at all and reports back.
-
@Ben Maddams Where is the above log from? What output are you seeing in each of the Monitor, Console Output and Debug Log tabs on the job itself?
-
3 minutes ago, Ben Maddams said:
Failed to post callback command 'log' to http://0.0.0.0:11117/_exec_callback/log/ No response was available.
@Ben Maddams The log has been very useful, thank you.
That 0.0.0.0 is going to be the problem. The code is making standard Windows API calls to get the list of addresses for the machine. We can override this in the config file, but would you mind checking a couple of things for me?
First, as admin on the machine running the SIS service, run
ipconfig /all
and post the results
Secondly, restart the SIS service and review the EspSisService.log file, located in C:\ProgramData\Hornbill\Site Integration Server\log. There will be a few lines in there - the one I'm interested in starts:
Starting HTTP responder on port 11117
and should contain a list of IP addresses at the end. -
@Ben Maddams The admin server (with the SIS service on it, I assume) and the DC won't necessarily be talking directly to each other, because the overall flow is
SIS Service <-> EspSisExec <-> (target specified in ITOM job) <-> domain controller
Some of those four could be the same machine, of course.
Can you share a screenshot of the job that is failing? -
@Ben Maddams Hello Ben,
It is not necessary to have the SIS service on the DC for any of the Hornbill Library packages. It can run on any server on the network, but ideally one that is joined to the domain you are managing.
As for your test jobs failing, the way the SIS service works is that when a job is run, an executable (EspSisExec.exe) is deployed to the machine specified as the target of the job, and it is this executable, rather than the SIS service itself, that does the heavy lifting and processes the package. While it is running, it reports back its progress to the SIS server, which then in turn reports activity to Hornbill, so that you can see the activity in real-time within the ITOM application.
This communication between EspSisExec and the SIS service is over HTTP, but using TCP port 11117, from the EspSisExec process to the SIS service. The error you are getting is because the SIS service is not receiving any communication from the EspSisExec process. The most common reason for this is an internal firewall, either somewhere on the network path between the two, or on the system running the SIS service, preventing traffic on that port reaching the SIS service. It's also possible, but much less likely, that there is some sort of firewall doing egress filtering on the system running the EspSisExec process. -
@Berto2002 Our cloud team have told me that they may be able to speed the deletes up for a period over the coming weekend, when the overall system load is lower.
-
@Berto2002 I'm not aware of anything in the development schedule to extend the functionality beyond the Deleted and Sent Items folders, so in this case, an enhancement request would be the way forward.
-
@Berto2002 For the first query, the delete is processed throughout the day, but it will not remove all the messages in one go. Are you seeing the count drop at all? It should be processing approximately 1,000 every hour.
For the second query, we do not automatically delete any emails not in Deleted or Sent Items, regardless of their age.
Error message running Jobs
in IT Operations Management
Posted
@Ben Maddams Hi Ben,
Just a quick follow-up on this. Are your jobs now running OK?